My game isn't lagging from WindowImpl::processJoystickEvents() any more (it hasn't for a while, actually).
However, I am finding that the majority of SFML time is still being spent in JoystickImpl::isConnected() and specifically inside the call to ClockImpl::getCurrentTime(). I haven't used joysticks at all yet, I'm just polling a RenderWindow in a loop.
This is what my stack looks like 20~30% of the time when I pause the game:
> sfml-system-d-2.dll!sf::priv::ClockImpl::getCurrentTime() Line 63 + 0x10 bytes
sfml-system-d-2.dll!sf::Clock::getElapsedTime() Line 49 + 0x13 bytes
sfml-window-d-2.dll!sf::priv::JoystickImpl::isConnected(unsigned int index) Line 58 + 0x22 bytes
sfml-window-d-2.dll!sf::priv::JoystickManager::update() Line 80 + 0x9 bytes
sfml-window-d-2.dll!sf::priv::WindowImpl::processJoystickEvents() Line 150
sfml-window-d-2.dll!sf::priv::WindowImpl::popEvent(sf::Event & event, bool block) Line 106
sfml-window-d-2.dll!sf::Window::pollEvent(sf::Event & event) Line 183 + 0x1a bytes
The variable Joystick::Count is 7 on my computer. That means that JoystickImpl::isConnected() would be called 7 times for each call to JoystickManager::update().