Since my last post I've switched over to pure OpenGL. I also found
this post. It suggests that when vsync is disabled, the CPU overloads the GPU with information, which have to catch up every second. I tested this out, but I'm sure this is false. There was also one guy mentioning that ProcessExplorer was affecting the game. I remember back when I was working with XNA, the refresh rate of Task Manager was influencing my game. I started killing background applications and once I killed
f.lux, my eyes hurt, but the largest lag spikes went completely away. In the end, the lag spikes was caused by external software, not SFML or graphics drivers. Although I'm still leaving SFML due to various other issues and limitations.