I think in SFML 2.3 we switched the audio implementation, which meant we run our own WAV reader.
Starting up the profiler on my side, the culprit seems to be the following line
while ((count < maxCount) && (static_cast<Uint64>(m_stream->tell()) < m_dataEnd))
More specifically the call to tell() which in turn calls std::ftell(), which seems to be rather slow.
For reading 10 relatively short WAV files, I see 8278 calls to tell() using up 84% of the time.
I wonder if it wouldn't be enough to just have the count and drop the tell() call entirely.
Edit: Removing the second check made everything load in the ms range again and so far nothing crashed. Looking at the OGG reader it seems we don't have that stream check there either and just do count < maxCount.
Laurent: Do you think it's safe to remove?
Created a PR, so feel free to comment there as well.