In theory you can use the source files as is, but you'll need to define all the necessary macros that CMake would set depending on the OS, architecture, etc.
VS projects always use "hard code paths". If you want the source to be somewhere else, you'd need to have it somewhere other than
C:/SFML.
Personally, I'd recommend to use the
SFML CMake Template, which integrates SFML as well, but you're working with a CMake project and the generated project files resemble your code base, while SFML is included without being overly noticeable.
While we've made sure to not do any CMake code generation, so the source could be used independently, we aren't actively supporting the inclusion of the SFML source in a project. Meaning, it's possible, but you'll have to figure out all the flags that are necessary to be set.