TCP Lag Reduction Techniques?

[binary1248]

It is a common misconception that you need to immediately handle incoming connection requests or pending data. If you don't do anything with them, they will just stay queued. And if your server loop operates at a decent framerate, it won't make a difference whether you handle them in the first possible frame or a few frames later. The player will not notice anything because anything under 20ms is barely noticeable anyway. The only time when it can get really laggy on servers is when they need to perform many database updates at the same time or read/write to disk a lot. Unless your server application is running at 100% CPU utilization I wouldn't worry too much about noticeable delays or freezes from the client perspective.

You must also be aware that your server as it is right now only handles a single game state, the "world" or whatever it is called. The data that represents this world is stored in data structures that are not meant to be used in a multithreading environment which forces you to use mutexes every time you want to alter it in some way. Since that is the only thing players do, you end up just blocking all threads except one. This is effectively a single-threaded server, just way more complicated. From my experience, I've seen servers employ threading in many ways. Most of them thread their database accesses because as I already said, at medium to high load, waiting for the DB calls to return takes too long to ensure a decent framerate. If you didn't notice yet, MMORPG developers also tend to separate their worlds into "zones" or "instances" in order to split the game state up into manageable pieces. I don't know the details but I'd say that it is a good idea to just have a single thread per game state. Any more than that and they would fight over resources to the point that you have no benefit and are left with the overhead of the extra idle thread.

[K-Bal]

Make it single-threaded, it will save you a lot of headaches and you get rid of your spikes

[Jungletoe]

I've seen servers employ threading in many ways. Most of them thread their database accesses because as I already said, at medium to high load, waiting for the DB calls to return takes too long to ensure a decent framerate. If you didn't notice yet, MMORPG developers also tend to separate their worlds into "zones" or "instances" in order to split the game state up into manageable pieces. I don't know the details but I'd say that it is a good idea to just have a single thread per game state. Any more than that and they would fight over resources to the point that you have no benefit and are left with the overhead of the extra idle thread.

I broke up the game data into "chunks". There is a single "world" in my game, but it's broken down into chunks for easier loading. Object and map chunks are 20x20 tiles. Each map tile contains 1 short, and each object tile contains 2 shorts. They're loaded through the C binary loading functions. I also load various player files and crate/storage data, but not on a per-chunk basis (both are handled on a per-request basis).

The thing is, I want the server to be able to handle 1000+ players easily. I don't see how I could single-thread this without it lagging to hell every time I get over 20 people concurrently online.

get rid of your spikes

How would that help? Wouldn't it just be running the same processes?

EDIT:
Wouldn't I also have to make it non-blocking?

[binary1248]

The reason it is locking up now is because of your threading model. Ironically, it would perform much better if you used a single thread to process your networking. You don't need more than 1 thread to handle rudimentary networking tasks.

If you check out http://en.wikipedia.org/wiki/C10k_problem you will see that there is a lot of widely known software that can handle 10000+ connections and the way they handle it is to either use asynchronous IO or some form of event-driven design. If they offloaded the network handling to multiple threads, the overhead would kill them at some point and that would make the software unscalable.

If you think that your server already can't handle 20 people concurrently, you should look into what the bottleneck is and resolve that before threading everything to oblivion. On the client side, the only common bottleneck is normally the rendering, but since your server only handles networking and game state updates it should have an easy time handling a large amount of players. If not, there is something wrong with your design.

If you read this: http://redmine.lighttpd.net/projects/1/wiki/DevelProblemAndSolution#Problem-5 you will see that the problems that the designer of lighttpd ran into are exactly those that I already mentioned before. That is probably the only place where you would need threading.

[Jungletoe]

Ok, one quick question before I actually go ahead and do all of this:
Is there an example of a non-blocking setup? I've seen the example of a blocking setup in the selector documentation, but I can't find one for non-blocking.

[binary1248]

No, unfortunately there is no example using a non-blocking socket. There is not much difference anyway. You just try to receive from the socket and if receive() returns NotReady, then you know that there was nothing waiting. I still recommend to use selectors even with non-blocking sockets. It might seem counter-intuitive but the selector strictly speaking only gives you a "hint" as to whether a resource is ready or not. In performance critical situations, it might just happen that a selector flags a socket as ready but you will still block for a small amount of time on the receive(). Since we want all the performance we can get we need to take this into consideration as well.

If you want to read more you can check this out: http://www.kegel.com/c10k.html

[Jungletoe]

Ok, I changed the networking around and people are having a lot of problems. Other people seem to have their clients not getting packets, and my packets that are sent to me are either distorted or not coming through.

Here is my receive packets function which is called every frame:

void NetworkManager::recPackets()
{
        Connector* newConnector = new Connector;
        newConnector->socket.setBlocking(false);
        newConnector->player = NULL;

        if(listener.accept(newConnector->socket) == sf::Socket::Done)
        {
                //      Adds a new client to client list
                maxConID++;
                newConnector->setID(maxConID);
                ent::connectors.push_back(newConnector);
        }

        else
        {
                //      Nothing we can do...
                delete newConnector;
        }


        //      The listener socket is not ready, test all other sockets
        for(sf::Int32 i = 0; i < ent::connectors.size(); i++)
        {
                sf::Packet packet;
                sf::Socket::Status status = ent::connectors[i]->socket.receive(packet);

                if(status == sf::Socket::Done)
                {
                        sf::Uint32 packetCode;
                        packet >> packetCode;

                        handlePacket(packetCode, packet, ent::connectors[i]->getID());
                }

                else if(status == sf::Socket::Disconnected)
                {
                        ent::playerHandler.handleDisconnect(ent::connectors[i]->getID());
                }

                else if(status == sf::Socket::Error)
                {
                        std::cout << "NetworkingError";
                }

                else if(status == sf::Socket::NotReady)
                {
                        //      Not ready
                }
        }
}
 

and my send packets function:

void NetworkManager::sendPacket(sf::Packet* pack, sf::Int32 connectorID, bool delPacket)
{
        if(getConnectorWithID(connectorID) != NULL)
        {
                Connector* c = getConnectorWithID(connectorID);
                if(c != NULL)
                        c->sendPacket(pack);
        }

        if(delPacket)
        {
                delete pack;
        }
}
 

Why are some packets distorted/missing and why can people from other computers not get all of the packets I'm sending them (I have no problem connecting with 2 clients at the same time on my PC besides the occasional distortion or packet drop).

[binary1248]

Your code looks fine to me, you can perform optimizations at a later time. Right now you need to check if the server is receiving data from the clients and correctly accepting incoming connection requests. If the server only sends data as a result of receiving data from clients, there are 2 possible error sources. Either it isn't receiving data correctly, or it isn't sending data correctly. From your code I can't tell which one (or maybe even both) it is. You need to either step through your code and see whether it does what you expect, or add some verbose std::cout debugging output which you can switch off later. You might also want to try out Wireshark to see whether the data corruption is happening in the server or during transmission. Unless you explicitly check what is in the Packets before you send them, there is no way to know whether they contain correct data or not.

[Jungletoe]

The thing is that all of the packets are being sent, just some aren't being received. I asked around, and it seems that the worse connection that somebody has, the more packets don't get sent to them.

Could bad connections cause packets to be dropped with non-blocking networking setups?\

EDIT:
I found this topic and it seems to have a similar problem... http://en.sfml-dev.org/forums/index.php?topic=3363.0
I don't understand what "polling" the socket means though.

EDIT 2:
So I think on the send() function, I should handle NOTREADY somehow? What should I do? I'm 99% sure that this is the root of my problems.

[binary1248]

If you use TCP then this should be taken care of at the transport layer. You shouldn't have to care about "lossy" connections. As long as some bits can get through TCP will make sure it recovers from errors eventually. This all occurs transparently and you as the application developer shouldn't have to care one single bit. If TCP determines that it cannot recover from an error it will just terminate the connection and SFML will notify you of this.

You can monitor this with Wireshark. You will see TCP retransmits marked as such if sending data over a crappy connection.

You also shouldn't forget that because TCP connections are full duplex, when you set them to non-blocking, it also affects sending. I don't know what your sending code looks like, but you are required to check the status code returned from the send() method as well. If it returns NotReady, you need to reattempt to send the data at a later time. If it returns Error or anything other than Done... well... you need to handle it as well .

[Jungletoe]

I don't know what your sending code looks like, but you are required to check the status code returned from the send() method as well. If it returns NotReady, you need to reattempt to send the data at a later time. If it returns Error or anything other than Done... well... you need to handle it as well .

THAT IS THE PROBLEM! I KNEW IT!

Okay... I'll try to fix this. I guess I'll put all "NotReady" packets inside of a vector and have the server cycle through it and attempt to resend it every frame.

[K-Bal]

The essence of this thread should be condensed and put somewhere into the SFML networking tutorials

[Jungletoe]

The essence of this thread should be condensed and put somewhere into the SFML networking tutorials

Hahaha agreed!

Anyways, with my new solution, it won't send the packets in order, resulting in a lot of problems. I'm going to have to put a std::list inside of the "droppedPackets" struct. I'll first check if the specific connector has any pending packets, and if it does, I'll add a packet into the list of packets it has to send first, that way it's always in order.

[Jungletoe]

Ok, having more problems. It's now sending the packets in order, but it's really buggy (it corrupts the packets). Also, the receive is also not working sometimes for those with bad ping and connection times.

How I handle dropped packets:

struct droppedPacket
{
        std::list<sf::Packet*> spack;
        sf::Int32 connectorID;
};

void NetworkManager::addDroppedPacket(sf::Packet* spack, sf::Int32 connectorID)
{
        bool found = false;
        for(sf::Int32 i = 0; i < droppedPackets.size(); i++)
        {
                if(droppedPackets[i].connectorID == connectorID)
                {
                        found = true;
                        droppedPackets[i].spack.push_front(spack);
                }
        }

        if(!found)
        {
                droppedPacket dp;
                dp.spack.push_back(spack);
                dp.connectorID = connectorID;
                droppedPackets.push_back(dp);
        }
}

void NetworkManager::retryDroppedPackets()
{
        for(sf::Int32 i = 0; i < droppedPackets.size(); i++)
        {
                Connector* con = getConnectorWithID(droppedPackets[i].connectorID);

                if(con != NULL)         //      client still online
                {
                        sf::Socket::Status status = con->socket.send(*droppedPackets[i].spack.back());

                        if(status == sf::Socket::Done)
                        {
                                //      packet sent correctly!
                                delete droppedPackets[i].spack.back();
                                droppedPackets[i].spack.pop_back();

                                if(droppedPackets[i].spack.empty())
                                        droppedPackets.erase(droppedPackets.begin() + i);
                        }

                        else if(status == sf::Socket::Disconnected)
                        {
                                ent::playerHandler.handleDisconnect(ent::connectors[i]->getID());
                        }

                        else
                        {
                                //      Packet did not send correctly
                        }
                }

                else    //      client disconnected
                {
                        while(!droppedPackets[i].spack.empty())
                        {
                                delete droppedPackets[i].spack.back();
                                droppedPackets[i].spack.pop_back();
                        }

                        droppedPackets.erase(droppedPackets.begin() + i);
                }
        }
}

bool NetworkManager::doesConnectorHaveDroppedPacket(sf::Int32 connectorID)
{
        for(sf::Int32 i = 0; i < droppedPackets.size(); i++)
        {
                if(droppedPackets[i].connectorID == connectorID)
                        return true;
        }
        return false;
}
 

How I handle the initial send:

bool Connector::sendPacket(sf::Packet* spack)
{
        if(!ent::networkManager.doesConnectorHaveDroppedPacket(ID))
        {
                sf::Socket::Status status = socket.send(*spack);

                if(status == sf::Socket::Done)
                {
                        //      packet sent correctly!
                        return true;
                }

                else
                {
                        sf::Packet* spack2 = new sf::Packet(*spack);
                        ent::networkManager.addDroppedPacket(spack2, ID);
                        return false;
                }
        }

        else
        {
                sf::Packet* spack2 = new sf::Packet(*spack);
                ent::networkManager.addDroppedPacket(spack2, ID);
                return false;
        }
}
 

How I handle receives:

void NetworkManager::recPackets()
{
        Connector* newConnector = new Connector;
        newConnector->socket.setBlocking(false);
        newConnector->player = NULL;

        if(listener.accept(newConnector->socket) == sf::Socket::Done)
        {
                //      Adds a new client to client list
                maxConID += 1;
                newConnector->setID(maxConID);
                ent::connectors.push_back(newConnector);
        }

        else
        {
                //      Nothing we can do...
                delete newConnector;
        }


        //      The listener socket is not ready, test all other sockets
        for(sf::Int32 i = 0; i < ent::connectors.size(); i++)
        {
                sf::Packet packet;
                sf::Socket::Status status = ent::connectors[i]->socket.receive(packet);

                if(status == sf::Socket::Done)
                {
                        ent::connectors[i]->logClock.restart();

                        sf::Uint32 packetCode;
                        packet >> packetCode;

                        handlePacket(packetCode, packet, ent::connectors[i]->getID());
                }

                else if(status == sf::Socket::Disconnected)
                {
                        ent::playerHandler.handleDisconnect(ent::connectors[i]->getID());
                }

                else if(status == sf::Socket::Error)
                {
                        std::cout << "NetworkingError";
                }

                else if(status == sf::Socket::NotReady)
                {
                        //      Not ready
                        //std::cout << "not ready";
                }
        }
}
 

Again, this (mostly) only occurs when the other connector has a bad connection and high ping times. Know of any problems with this?

[binary1248]

What do you mean by "corrupts the packets"? If you mean data corruption, how do you determine this? Did you check whether this corresponds to what the server is sending?

As for the receive not working, it might also be the client code that is not sending/functioning properly. If you want to make robust software that functions even over bad connections, you need to make sure that both sides are able to handle such a scenario.

Maybe you could show the client send and receive code as well.