Multiple files only decrease compile if you carefully check dependencies, and possibly use the PImpl idiom. If you have a #include <SFML/Graphics.hpp> + several standard includes in every header file, your code is almost guaranteed to compile slower than doing the same thing in a single header file. After all, the compiler needs to re-compile the same code over and over (unless you use precompiled headers).
You can definitely start with one file, C++ leaves it up to you how you organize your code. But once your code grows beyond a few hundred lines, it really gets harder to navigate. At some point, compile times do come into play, because that one file (and thus your entire code) needs to be recompiled on every change, no matter where.
Consider this: you have files Zombie.hpp and Zombie.cpp which implement a class. The header is simply the public interface + some member variables. The logic how the zombie moves is written in the .cpp file. Now if you want to modify something, e.g. make it faster to move or change how it interacts with certain objects, the compiler only has to recompile Zombie.cpp + all the headers it includes. But it does not have to recompile Player.cpp, Item.cpp and Orc.cpp. They are independent translation units.
