There isn't a singular cause. Anti-virus software uses lots and lots of different heuristics and code signatures. Whether your binary triggers such a heuristic is impossible to say or predict.
Signatures might match with some tool that used the same Windows API as SFML and that might already be enough.
Avoiding sf::Keyboard::isKeyPressed() and potentially sf::Mouse::getPosition() are a good idea, because they use "global" APIs and aren't restricted to a specific window, as such are often used for "spying" on the users. Which is why on macOS you need to explicitly granted monitoring access to the application to be able to utilize them and Wayland doesn't even directly offer certain functionality.
If you want your applications to not be flagged, you may submit it to virustotal and open false-positive reports with the AV companies that wrongly flag it.
If you're really serious about this, you can sign your application with a certificate, but this usually requires some sort of legal entity and money.