Thanks, that is interesting and helpful. If SFML is so modular that this is possible that would be great. However, this time I was able to build and install the necessary audio libraries in my home directory, and have now also successfully compiled and installed a git snapshot of SFML, so everything is working fine now.
I might have to compile SFML again some time, and then I will try to see if I can do without any audio libraries.
Thanks!