You must be drawing a lot of texts as I've recently been doing some tests and failed to gain any substantial speed from batching many texts into a single vertex array.
With that said, however, the test was with identical character sizes.
With different character sizes, a different texture needs to be used.
So, using multiple character sizes means multiple textures and each time you draw a different size, you are switching that texture (the expensive thing!)
One thing you can try is to draw all texts with the same character sizes at the same time; this should reduce texture switching and allows sf::Text to work some magic.
Note that switching fonts is also texture switching so to 'not switch', it must be the same font
and the same character size.
You can still scale a text without texture switching but this is often less than desirable.
As an aside, I'm unaware if my tests were unsuccessful just because of my hardware (or some other optimization or maybe some other reason) so I may continue to develop a text batcher