moving GPU to CPU is always slow, unfortunately.
but in this case you could try an OpenGL function, like glReadPixels.
unsigned char data[4];
glReadPixels(sf::Mouse::getPosition(window).x, sf::Mouse::getPosition(window).y, 1, 1, GL_RGBA, GL_UNSIGNED_BYTE, &data);