Whenever you run into an issue that might be traced back to SFML, you should go grab the latest version and build SFML from that, just to make sure nothing got fixed in the mean time.
Before or after that make sure your audio driver is up-to-date.
As for the big latency before it starts playing, could you attach a profiler and look where the time is being spent?
Also make sure the OpenAL version that ships with SFML is used and you don't have another version for example in the system32 directory.
For me it however largely sounds like a driver issue.