Well, problem is how extensive SFML makes use of the GPU to increase performance.
I don't know about that issue, but if SFML's increase in performance over SDL is heavily GPU-related and it is not that optimized for CPU, maybe SDL has a better rate on CPU-usage only.
However, I'm not really on the inside of SFML, so I can only guess which API is slightly more performant, using a VirtualBox.