Well, this one's definitely different. The debug build of my game runs at the correct 65 fps. The release one will either run at 50 or 32. If I remove the framerate cap, it naturally flies, so no way it's a memory issue...
I naturally tried commenting out things, but nothing helped.
All the game even does is move the player around. I could post the code for while the game is running unless someone has a good idea what it could be.
UPDATE: I actually re-implemented my old idea of having a time check at the end, rather than regulating the framerate to be 60 and this fixes the problem... I don't know if this is the best fix, but for now I'll take it.