Ended up fixing this problem a while ago but forgot to update this question. I ended up creating a Camera class which handles sf::View and then translates anything drawn by opengl by how much the sf::view has moved as well (almost like having 2 separate cameras).