Alright, thanks for the help! Views it is.
I ended up using:
sf::View view;
view.reset(sf::FloatRect(0, 0, 320, 240));
view.setSize(320, 240);
window.setView(view);
Or more generally:
sf::View view;
view.reset(sf::FloatRect(0, 0, map_width, map_height));
view.setSize(map_width, map_height);
window.setView(view);
As I draw my sprites relative to 0, 0 and the edge of the scene is 320, 240.
If I understand correctly, I first reset the View to 'focus' on the (0, 0, 320, 240) of the whole scene, and then I set its size to 320x240 so it shows the whole (0, 0, 320, 240) rectangle (from the 'focus') on the View's viewport which is (0, 0, 1, 1) i.e. the whole screen. Thus I get the (0, 0, 320, 240) to the whole viewport. And all is scaled as I wanted.