If SFML was a hidden implementation detail of your library, your executable wouldn't have to include or link anything about SFML (unless your library is a static one).
But, since SFML appears in the public interface of your library, the executable indirectly uses it. In particular, it indirectly includes some SFML headers. So it has to know where they are, so you have to configure your project accordingly.
But this is a design issue, not a technical one. Either you want SFML to be a hidden detail, and you do whatever is required to actually hide all SFML classes and calls, or it's really part of your public API and you accept that your library is not independant, it's a SFML add-on, which requires SFML to be used.