I'm not sure about compiler time, but that is probably the case.
The size issue is a good point in certain circumstances. If you have one SFML program on your computer then it doesn't make much difference whether the SFML code is in a shared file or inside your program, but suppose you have five SFML programs. It's obviously better for them all to share the same SFML code than for each to keep its own identical copy.
One particularly good example of this is the C runtime. Imagine if every program you made in Visual Studio had the Visual C++ runtime code in it. It would be crazy ^^
Also, some licenses (e.g. LGPL) only allow you to link dynamically.
