Yes, it's just compile times. And it also speeds up tools that perform static code analysis (auto completion, real-time error messages, dependency graphs, doxygen, ...).
(There can be rare cases where it actually affects runtime behavior, such as static/const variables in headers that are not optimized away, symbols that participate in ADL or overload resolution, or macros that change the compilation. But the primary goal is to reduce compile times.)