From my basic understanding of sfml's networking library and networking in general, TCP requires a connection while UDP does not. When you call the send command for TCP it tells the other end that there is a packet coming, waits for a reply, then sends the package, waits for conformation that the package was received, and then continues with it's business. Due to TCP requiring a connection between the client and server you often need to have multiple sockets (usually one for each client).
UDP, on the other hand, often just sends the packet without notifying the receiver that it is going to send a packet. Due to this lack of connection between the server and client UDP only requires one socket to receive and send data.
The reason socket Selectors exist is to prevent socket blocking (mainly with TCP) from becoming a problem. With the socket selector you can poll the status of and receive from many sockets without having to wait on any one socket to receive data. Using UDP with a socket selector would be pointless because there's only one socket (the receiving and sending socket) meaning there aren't any other sockets to block. Because of this, you should probably just store the IP address of each client since you can't make a socket for each client with UDP, you can only make one for the receiving and sending end of the server.
TLDR you should just store the IP's of the clients.