I would assume that you care about performance, because OpenGL is well supported, and it isn't outdated. In this case you will be always better(quicker) when wrting in pure OpenGL/Vulkan (assuming you know what you are doing and write code super optimally) instead of using SDL2/SFML/any_other_lib.
Maybe something specific from SDL/SFML that would optimize code for someone is slowing down program in your case, maybe you could optimize something that those libraries doesn't consider, maybe you could (in your case) avoid some boolean checks, because you need more "specialized" solution and don't need those bool checks.
I sometimes look into SFML/SDL implementation files when I use them and I see "oh, this code is something I don't need, but I know for what cases it's here". Heck, sometimes I even rewrite something too, or inherit from this stuff, because why not...
If you are so concerned about performance, than perhaps you should be use neither SDL2 nor SFML? use Vulkan and implement everything yourself. Or use SFML and If you have some performance bottleneck in some specific place, than you can somehow implement this part in OpenGL/Vulkan yourself...
