I have exactly the same issues. Here's what I found :
- Deactivating ButtonMotionMask and PointerMotionMask in WindowImplX11.cpp removes bad performance caused by the motion of the mouse, but also disables MouseMove events. It's still possible to retrieve mouse position thanks to sf::Mouse::getPosition(). A better fix would be to offer the user an option to sync internal mouse event retrieving with display() (thus getting the mouse position with sf::Mouse::getPosition() and putting it into memory so that it can be used later by pollEvent()). This way, only one mouse event would be retrieved per frame. Actually, this is option is granted by many recent STR games : you usually have an option to "reduce mouse latency", which makes the game behave like SFML : less mouse latency, but bad performances on low-end computers with high-resolution mices. (Actually, I think many of these games still limit the amount of mouse events retrieved per frame, since the framerate isn't SO bad.)
- The way joysticks are handled is really an issue. I also have an accelerometer incorporated in my computer, and this is also causing slowdowns. it's because of the way joysticks are detected. Every time pollEvent is called, the list of joysticks is updated, and SFML tries to find if new joysticks were plugged in. This detection is performed updatePluggedList in Linux/JoystickImpl.cpp. When this function tries to know if /dev/input/js0 is plugged in, it calls the syscall "stat" to know if /dev/input/js0, and it actually exists, even if the user doesn't have the rights to open it. Then JoystickImpl::open calls the syscall open("/dev/input/js0") even if it doesn't have the right. And I think this is this syscall that takes time.
- If SDL and Allegro don't have the same issue, it may be because they us XInput instead of using directly the /dev/input files, because XInput KNOWS it cannot use this "joystick" (=internal accelerometer). But I have to read the sources of SDL to be sure they really use XInput.
I think the first thing to do is to let the programmer choose if SFML should try to use Joysticks, because even if joysticks may work fine on many computers, SFML still calls "stat" Joystick::Count time for EVERY pollEvent call on these computers. I think this is bad because it uses the hard drive, which is a slow device (well, i must admit i don't know how "stat" works, maybe it's optimized and it doesn't use the HDD for every call, but still...).
As you said, the best way to solve this may be to look at the way SDL finds joysticks.
Hope this helps, and thank you for having created this amazing library !
EDIT : Appearently, SDL doesn't use XInput to handle joysticks. I'm currently trying to understand how it's done.