Alright I've tried everything now. I've tried compiling with static, debug, release, shared, on a different machine, every possible combination. At the first reference of SFML graphics specific code is about to be run it freezes. So I'm guessing that something funky in graphics context creation is being done. Why it doesn't happen in rbSFML is a mystery to me.
Should I post this as an issue?