Compilation speed is about the same: around 1 second for each file, so any difference is hardly noticeable.
There are some headers which have a lot of dependencies on their own, so it will be difficult to notice.
What also has to be taken into account, is that standard library headers take rather long to parse, because they use loads of templates (for Boost, it's even orders of magnitude worse). Thus, if you can avoid including a SFML header with its
<vector> and
<map> includes, that may already make a difference.
But don't exaggerate just to get some milliseconds; if you follow eXpl0it3r's advice of only including the necessary headers, it should compile fast enough. When you get into problems, the Pimpl idiom is also an option to reduce compile-time dependencies.