Depending on which version I decide to use.
Which would be more stable?
Use SFML 2, because it's more stable and it's maintained. SFML 1.6 hasn't been touched for over 2 years, has many bugs (e.g. the very very bad ATI bug) and lacks a lot of nice features (VertexArray, RenderTexture, ...).
I saw there are A LOT of tutorials for 1.6.
Yes, SFML 2 doesn't have all the tutorials, but it's mostly quite easy to port the 1.6 ones to 2.0.
If I decide to build 1.6, where do I find the sources?
Also ... I need to have CMAKE installed right?
You'd find the source on GitHub in the branch sfml1 and no you wouldn't need CMake, since SFML 1.6 doesn't build with CMake. You'd need to convert the provided VS project file to VS2012 and hope that it works...