I ended up with the same issue of the stuttering returning after rebooting, even with vJoy entirely uninstalled and my physical joystick unplugged. At that point I tried getting the peripherals down to only my mouse, keyboard, and headset. That still didn't work. At that point I ended up launching the Logitech configuration software I use for my mouse and headset, and that seemed to have reduced the lag (but it still exists).
To give details regarding this, the lag from the polling happens in approximately 500ms intervals. I would be curious if the interval is similar for you, Ushi.
At this point I'll try building 2.4.1 without the joystick polling and report back the results.
Edit: Ran a bare-bones example through the profiler after compiling SFML 2.4.1 to get the debug symbols. This definitely stems from the joystick polling, even when all of my joystick peripherals are disconnected. The call to getJoyPosEx in the Win32 JoystickImpl::isConnected seems to be the culprit, which seems to be a (partially?) known problem given the comment in the function and the implemented workaround (which is to implement a 500ms delay between checking the joystick connection, which explains why I get the stuttering every 500ms).
Here's a link to the specific line on GitHub for reference.Will update again if I figure out a possible solution. Obviously (as mentioned earlier in the thread) compiling a version with joystick support disabled would work, though it sounds like that's not going to work for you, Ushi.
Edit 2: Looked around a bit longer out of curiosity (I just disabled joystick event polling, which worked as eXpl0it3r said it would). From what I garnered from some Google searches, WinMM, the library that SFML uses for handling joystick input,
is deprecated in favor of DirectInput. It might be possible that there are some problems running on Windows 10 systems? I don't really pretend to know and I'm hardly an expert, but I thought it might help out if you want to narrow it down.