I thought there would be an easy solution!
There is; render to a texture first

I would like to write my own implementation [...] with the render to texture, but as my OpenGL is a bit rusty, it will probably take a while.
Create a render texture of the size of the amount of map you want to show when it is not zoomed or moved (no fractions of pixels). Render to it without any scale and make sure all positions are integers (assuming its view is 1:1. Create a "render sprite" to show the render texture and set its texture from the render texture. Draw the render sprite directly to the window.
Probably look something like this:
sf::RenderTexture renderTexture;
sf::RenderWindow window;
// .. set up render texture and window (sizes and views if required, for example)
while (window.isOpen())
{
// event loop
// update loop
// render to render texture
renderTexture.clear();
renderTexture.draw(tilemap);
renderTexture.display();
// prepare render sprite
sf::Sprite renderSprite(renderTexture.getTexture());
renderSprite.setScale({ 2.f, 2.f }); // zoomed in x2 (to top-left corner as we haven't adjusted its origin)
renderSprite.setPosition({ -31.24f, -5.91f }); // just a non-integer number for position offset
// render to window
window.clear();
window.draw(renderSprite);
window.display();
}
You may find the sf::RenderTexture documentation useful here:
https://www.sfml-dev.org/documentation/2.4.2/classsf_1_1RenderTexture.phpEmbarassing to say it's simply a sf::clock which is created in main and then passed as clock.restart() to the update function of my Game class. The usage will then be to multiply with the passed deltatime (dt.asSeconds()). I also tried Kairos' timestep but it only made the problem worse, perhaps could have sometime to do with the fact that I'm using box2d for physics when moving the player.
Kairos' Timestep breaks up time into equally sized pieces so dt is identical on every update frame. That's its whole point.

Notice that if the amount of time passed doesn't perfectly divide into dt-sized pieces, there will be a slight difference between the actual time passed and the amount of time processed by the updates. This is where the interpolation comes in. Basically, you take the previous state (all positions, sizes, velocities etc.) and the current state and then interpolate between them using the interpolation from Timestep. This creates an exact one-frame lag but motion becomes smoother and is accurately following time.
This method is described in
Fix Your Timestep under the heading "The Final Touch". Note that the fixed dt done by Kairos is implementing a method based on the "Free The Physics" part of this article. It's up to you to use the interpolation (it's calculated for you - timestep.getInterpolationAlpha()) to add the final touch.
Here is a simple example on how to use interpolation and how it affects thing.
Here is a video showing that example in action.
If Kairos is doing anything strange (not described above), it needs fixing!
