Depends on your specific frame buffer device.
SFML supports Direct Rendering Manager (DRM), but it requires SFML to be compiled for it.
There's currently no dynamic switching between the two.
In order to use it, build SFML with the CMake option SFML_USE_DRM enabled.
This way, you won't need X11.