The problem is that the semantics of your class are not well defined. It is copyable (and it is copied when you add instances to the vector), but the copy operation is not properly implemented: several copies of the same player will share the same socket handle, but whenever one of those players will be destroyed, this handle will be closed and the all other copies will report a disconnection.
So you should ensure that your Player class should never be copied (make it inherit from sf::NonCopyable for example) and store pointers in your vector.