EDIT
Solved my problem, which was because I was accidentally misusing and the question (Lots of small packets or one big one) I just decided to use medium sized ones, since it seems to have the best of both worlds. I'll probably split each thing up and give it a packet (Ex one for the map changes, one for entities, etc)
The reason it was going wrong was because I was disconnecting a client that had already been disconnected. I have no idea what I was thinking at the time ^_^, and a second reason was I was trying to move something while iterating on it. ¬_¬
client.disconnect();
networkSelector.remove(client);
clients.remove(&client);
Was my code. I then removed first line and modified the removing system clients and it seems to be flawless with connection and disconnection.
Sorry for posting since I ended up solving it myself.