Hmm, well using log messages pointed me towards the glGenBuffers line, but that seems to be unreliable, since it happens there sometimes but not always. Unfortunately the debugger gives no useful information, except that the crash happens somewhere in the video driver.
Disabling VBOs completely and using immediate mode instead is the only thing I've found so far that fixes the problem, but of course that isn't really a satisfactory solution.
Testing on my laptop with an intel chipset instead of nvidia like my pc, in release build it locks up, but in debug mode it works but with lots of graphical artifacts on the things drawn using VBOs. Is there some kind of VBO-related state that SFML 2 is setting?