What do you exactly mean by panning?
I would guess that panning in this context means changing the stereo positioning (left to right). The spatial positioning looks complicated if you just want stereo panning.
change the sf::Sound's panning without to use the sf::Listener ?
I think that the sf::Listener is getting used whenever you play any sounds so I don't think you will have to actually change sf::Listener. It should be okay to use sound.setPosition() to set the panning. The default position of sf::Listener is (0, 0, 0) so setting the sound's position to (1, 0, 0) would give you "right" and (-1, 0, 0) would give you "left". Note that those are hard pans though and I think that any positive number (x, 0, 0) would be hard right. If you need part panning, you should use adjust x and z together to. e.g. (0, 0, -1) would be in front (centred panning).
I haven't done enough with this stuff yet. I'm going to go and mess with it
If I make a panning function (I can't imagine it'll be hard), I'll let you know
Still, I'm sure you can figure it out