I can provide minimal code if anyone needs it, however it is currently 1:15am and i really need to sleep!
If you've been trying to solve this all day and still don't have a minimal example, then you've been doing it wrong.
To test what has which effect from where on what, you always need to start cutting down and off or go the easier way and start new with the lowest number of things, i.e. a normal game loop and a way to tell the fps.
With that you can test how the FPS is, without a sound object and how it is with. You can then also run profilers, to potentially figure out where the application spends a lot of unnecessary time.
Are both, your audio and graphics driver uptodate?
Do you notice any problems when playing a sound with another application?
My best guess would be, either you're handling SFML's "sound system" wrong, or you might have a problem with the given OpenAL version.
What SFML version are you using (i.e. from where did you get it)?