Linux is usually the system where dependencies are the easiest to manage. You don't have to distribute anything, users just have to "apt install libflac" or equivalent (which will most likely already been done, since flac is such a common dependency for anything that deals with audio).
Personally I'm not against making things optional, but they have to be worth it. And don't forget about the drawbacks: it introduces clutter at configuration time (I'm currently working with a library with tons of CMake options and I'm still trying to figure out which ones I should enable or disable), and most of all, all the SFML libraries built with custom options will generally be incompatible with each others, making the distribution process even harder.
Regarding audio back-ends, since the current system is quite modular, maybe we could think of runtime solutions rather than configuration options -- something like plugins.