Plinth without SFML would be the base part i.e. just the stuff in Plinth directory (built as a library).
Plinth with SFML would require both the base part and the stuff in the SFML directory (built as a separate library).
If you use any of the SFML parts yourself, you need to include SFML yourself, yes.
If you use Plinth's SFML module, you need to link in SFML as well.