I hope a one-time double-post is okay to bump the thread.
http://en.sfml-dev.org/forums/index.php?topic=8970.msg60382;boardseen#new
This thread just helped me narrow it down to the event-handling.
If I remove the event-loop (and reboot the system) the probelm doesn't occur anymore.
(I wonder why I never thought of that before)
Quite interesting to see, but I'm still as helpless as I was before.
[edit]
Now that I know it's linked to events I can tell you that the probelm is present either in bigger apps that make heavier use of events and also testing apps that only contain event-loops like this:
sf::Event event;
while (window.pollEvent(event)){
if (event.type == sf::Event::Closed)
window.close();
}