Wow, OK, so this could be a difficult one to untangle.
The first thing is that SFML is not building when you build your code. You're using a library that's already built - so SFML is *always* in Release mode.
That means it's probably your code. And it would DEFINITELY be a good idea to test it out on different machines in both Release and Debug modes.
If you haven't already, it's worth getting SFML to use VSYNC, or force it to 60FPS (or whatever FPS you will find acceptable.) Before you publish, you'll want to consider whether to support machines that are too slow to run at 60fps - which could give you major headaches if your game logic is dependent on specific numbers of frames passing. 60fps is roughly 1 frame per 16.6667ms (or 1/60 of a second) so you want to either do all your processing in 16ms or force your framerate down to the number of milliseconds you need to do your inter-frame processing.
I'm interested to know - if you don't consider yourself a *fantastic* programmer, why do you use C++? C# is far more forgiving. You don't need to worry about allocating memory, freeing it, and all that. I'm a bloody amazing coder (although a really bad mathematician), and I prefer C#. And yes, it is definitely fast enough.
Now, I don't know what your code looks like, but if you're creating a new Time object frequently, that might be causing you problems. For keeping track of time, you're better getting the start time of your application then using Modulo (% operator) to get the current value of a hypothetical looping timer. Let's say a is always increasing, a % 100 will always be between 0 and 99, and will "roll over" to 0 when it reaches 100.
For moving stuff around I also prefer to 'outsource' the responsibility to a physics engine. I'm using Box2D/LiquidFun. Main issue I have at the moment is that it's single-threaded, and making it multithreaded would not be a straightforward project. So I just call Box2D in a separate thread. I have a semaphore joined to my vsync, so Box2D runs in parallel to my game logic. I've got a C# wrapper around a C++/CLI wrapper around Box2D with more semaphores so that stepping the world and modifying the world do not overlap. That's not to say that this is necessarily a "normal" way of doing things - you don't have to follow my example. I am not a professional game developer (yet).
The other thing you might consider is offloading your starfield to GLSL. Using a Stopwatch (or just counting milliseconds since your exe started) you can pass the number of elapsed milliseconds to a Fragment Shader, and let that deal with drawing a starfield. If it's a horizontal startfield you could "take inspiration from" these:
http://glslsandbox.com/e#41903.0http://glslsandbox.com/e#41888.3