Well indeed what do you know. I've re-opened up my OGG in Audacity, normalized it to -1.0f, re-exported and it now plays back flawlessly... Doing this for every one of my music files would be highly inconvenient though, as stated on the issue page.
What makes me wonder though, is that other players like Audacity itself and VLC were able to play the OGG back flawlessly. I highly doubt that they normalize it themselves. If, they however do it, they have found a way to do it efficiently without hogging the CPU, so wouldn't this also be possible for SFML?
EDIT: I've just read that the issue lies in a conversion issue from libsndfile itself. Is it maybe an idea to use libvorbis instead for OGG music? It would require extra code; a special case for OGGs, and an additional dependency though. Or perhaps use sf_read_float in the special case of OGGs, as that apparantly seems to work.