Turns out I had some old SFML libraries in /usr/lib. So getting rid of those fixed it.
And about the order, I've found it funny my system seems to be okay with doing it the wrong way. It works, and trying the executable on other systems works as well, so I'm not complaining. (I normally do it the right way, just typed it wrong here.)
Thanks for the answers.