Hard to say since every compiler will add its own stuff. But in general debug libraries get debug symbols, so that the debugger knows what belongs to what, additionally the code doesn't get optimized (too much).
For example if you run the debugger with release binaries and try to break at certain locations, it will just jump over them and break somewhere else, because either the code piece got optimized away or the debugger simply doesn't know where to stop.
The linker issue one sees with VS is mostly always due to the fact that, when you're in debug mode VS will use the debug runtime libraries, while the SFML DLLs are linked with the release runtime library and thus there are somewhere incompatible functions. Interestingly enough most of the time it will cause an error with strings, thus after seeing it a few times, the issue gets obvious quickly.
