Welcome, Guest. Please login or register. Did you miss your activation email?

Author Topic: TCP Lag Reduction Techniques?  (Read 22355 times)

0 Members and 1 Guest are viewing this topic.

Jungletoe

  • Full Member
  • ***
  • Posts: 132
    • View Profile
    • Email
TCP Lag Reduction Techniques?
« on: March 30, 2013, 09:14:30 pm »
Hi, I'm developing an sandbox ORPG called Colonies.

As my game has progressed, we have gotten upwards of 20 players constantly on at once. With this new traffic, there is also a lot of lag that has followed. After optimizing everything except the networking backend, I came to the realization that the lag was coming from sending packets.

The game can suddenly freeze for 5-20 seconds at worst. I profiled the threads and noticed that it was getting held up at "zwwaitforsingleobject" inside of the send packet function.

-What are some common techniques to reduce the sending lag?
-Would breaking down the packets into smaller ones reduce the lag or have no effect since SFML does this anyways (I've noticed that larger packets take longer to send)?
-Does adding a timeout to the selector effect the timeout of the send? Do I need to have the selector in a different thread from the "sendPacket" functions in order for the timeout to work?

Thanks,
Jungle

Varonth

  • Newbie
  • *
  • Posts: 11
    • View Profile
    • Email
Re: TCP Lag Reduction Techniques?
« Reply #1 on: March 30, 2013, 09:42:53 pm »
May I ask what exactly are you sending there?
5-20s sounds like you are sending something huge.

Jungletoe

  • Full Member
  • ***
  • Posts: 132
    • View Profile
    • Email
Re: TCP Lag Reduction Techniques?
« Reply #2 on: March 30, 2013, 10:13:25 pm »
May I ask what exactly are you sending there?
5-20s sounds like you are sending something huge.

Map chunks (about 20x20 ints), object chunks (can be anything from 0 ints to 30x30-- although rarely that high), and player movements (player name string, int x, int y, bool running) in which a for loop adds multiple player positions to the same packet if the player has moved.

The player movement packets take the longest to send if there are 2+ players all putting in a movement request at once, which is where the problem arises.

Varonth

  • Newbie
  • *
  • Posts: 11
    • View Profile
    • Email
Re: TCP Lag Reduction Techniques?
« Reply #3 on: March 30, 2013, 10:35:22 pm »
Int in like 32bit standard integers?
Because one of these 20*20 packets would be 20*20*32/8 = 1600 bytes.
The 30*30 packets would be 30*30*32/8 = 3600 bytes.

Also you are sending the character name with each move?
If so, that is alot of bandwidth wasted.

If you need a way to distinguish the different move request for each character, you should consider using a sf:Uint8 or sf:Uint16 as an ID number, and give every single player a unique ID for that playsession, which allows every client to know which character entity is moved by each move request.

You should try to make packets as small as possible, as 1kb packets send to 20 people, perhaps multiple times per seconds, takes alot of bandwidth pretty fast.

And while most home connections have a good downstreams, the upstreams are often limited in speed.

Jungletoe

  • Full Member
  • ***
  • Posts: 132
    • View Profile
    • Email
Re: TCP Lag Reduction Techniques?
« Reply #4 on: March 30, 2013, 10:40:15 pm »
Int in like 32bit standard integers?
Because one of these 20*20 packets would be 20*20*32/8 = 1600 bytes.
The 30*30 packets would be 30*30*32/8 = 3600 bytes.

Also you are sending the character name with each move?
If so, that is alot of bandwidth wasted.

If you need a way to distinguish the different move request for each character, you should consider using a sf:Uint8 or sf:Uint16 as an ID number, and give every single player a unique ID for that playsession, which allows every client to know which character entity is moved by each move request.

You should try to make packets as small as possible, as 1kb packets send to 20 people, perhaps multiple times per seconds, takes alot of bandwidth pretty fast.

And while most home connections have a good downstreams, the upstreams are often limited in speed.

Hmm ok... I already have IDs for animals, so Ill just switch that code over to the players. That should be a simple fix.

And yes-- the 32bit ints.

So should I break down these packets? Would that solve anything? And is it just due to the fact that Im hosting it from my house network, because games like Minecraft don't seem to have that problem with 11 people.

Varonth

  • Newbie
  • *
  • Posts: 11
    • View Profile
    • Email
Re: TCP Lag Reduction Techniques?
« Reply #5 on: March 30, 2013, 11:01:35 pm »
I don't know how your code works exactly. For example I don't know how often you send the map and object data, who gets which data etc.

If it is just once per second per client, with the numbers above this would be 5200*20 = 104000 bytes in a worst case scenario. That is 101.5625 kbyte/s or in terms of upstreams, 812.5kbit/s, just for this data. If you map/world can be indexed with 65,536 different states you might also switch to sf:UInt16 for your map and object chunks. But would basically halve the size of these sends (not actually 50% because of overhead by Tcp and IP but it would be so close, that you could say it is halved).

On top of that you might consider data compression before you send the data.
The packet documentation has some informtions on that topic, and there are several open source libraries for compression, like zLib.

It is probably one of the best advices I can give to you. Keep your packets as small as possible.
You often have to send this data multiple times, sometimes (or depending on the game always) to everyone connected to the game. Saving 20 bytes of data in a packet ends up being hundreds of bytes that you don't have to send.

Jungletoe

  • Full Member
  • ***
  • Posts: 132
    • View Profile
    • Email
Re: TCP Lag Reduction Techniques?
« Reply #6 on: April 02, 2013, 03:16:59 am »
Ok, I reduced the size of the packets. Still no luck. The server still pauses/freezes for 10-30 seconds on ZwWaitForSingleObject.

binary1248

  • SFML Team
  • Hero Member
  • *****
  • Posts: 1405
  • I am awesome.
    • View Profile
    • The server that really shouldn't be running
Re: TCP Lag Reduction Techniques?
« Reply #7 on: April 02, 2013, 05:03:56 am »
I really want to help, but without more insight into how your networking code looks like, I'm limited to making wild guesses and possibly providing unhelpful advice.

Maybe it might be a bit nit-picky but what you describe can't be classified as "lag" the colloquial term for network latency. Network latency is the the concept of the time that is needed for data to be transferred from one host to another. Low latency is good, high latency is bad for real-time applications but can be acceptable for bulk data transfers. If your server locks up for 10-30 seconds it isn't due to latency because no data is arriving at all at the clients in that time interval, and when it starts sending data again it all arrives at once.

From tests I conducted while I was very bored once, I can say that SFML doesn't have any significant network bottlenecks, especially at 20 connected clients. Therefore I am relatively sure it is something specific to your code/architecture. Because of this, if you expect useful advice, you need to provide more specifics other than the kind of data you send. The amount of data you send is also insignificant, otherwise you wouldn't be able to host Minecraft with 11 people. Making use of multiple threads is also a headache to debug as you found out. There are ways to do it but none of them are really reliable if you ask me, especially on Windows. WaitForSingleObject is a win32 synchronization function that is probably called when you make use of Mutexes or block on something.

Maybe you could provide code snippets. They would provide more information than the questions in your first post.
SFGUI # SFNUL # GLS # Wyrm <- Why do I waste my time on such a useless project? Because I am awesome (first meaning).

Jungletoe

  • Full Member
  • ***
  • Posts: 132
    • View Profile
    • Email
Re: TCP Lag Reduction Techniques?
« Reply #8 on: April 02, 2013, 07:08:10 am »
Maybe you could provide code snippets. They would provide more information than the questions in your first post.

Hmm ok. What snippets do you need exactly?

Here is my server-sided "receive packets" function:

void NetworkManager::recPackets()
{
        // Receive a message from the client
        while(running)
        {
                //      Wait for data on any incoming socket
                if(Selector.wait(sf::milliseconds(250)))
                {
                        //      Test the listener
                        if(Selector.isReady(listener))
                        {
                                maxConID++;
                                Connector* newConnector = new Connector;
                                newConnector->player = NULL;
                                newConnector->setID(maxConID);
                               
                                if(listener.accept(newConnector->socket) == sf::Socket::Done)
                                {
                                        mu::lockMutexes();
                                        mutex.lock();
                                        //      Adds a new client to client list
                                        ent::connectors.push_back(newConnector);

                                        //      Add a new client to the selector so we will be
                                        //      notified if we recieve something from it
                                        Selector.add(newConnector->socket);
                                        mutex.unlock();
                                        mu::unlockMutexes();
                                }

                                else
                                {
                                        //      Nothing we can do...
                                }
                        }

                        else
                        {
                                //      The listener socket is not ready, test all other sockets
                                for(sf::Int32 i = 0; i < ent::connectors.size(); i++)
                                {
                                        sf::TcpSocket &client = ent::connectors.at(i)->socket;
                                        if(Selector.isReady(client))
                                        {
                                                //      The client has some data, we can recieve it
                                                sf::Packet packet;
                                                if(client.receive(packet) == sf::Socket::Done)
                                                {
                                                        sf::Uint32 packetCode;
                                                        packet >> packetCode;

                                                        mu::lockMutexes();

                                                        handlePacket(packetCode, packet, ent::connectors[i]->getID());

                                                        mu::unlockMutexes();
                                                }

                                                else if(client.receive(packet) == sf::Socket::Disconnected)
                                                {
                                                        mu::lockMutexes();

                                                        mutex.lock();
                                                        ent::playerHandler.handleDisconnect(ent::connectors[i]->getID());
                                                        mutex.unlock();

                                                        mu::unlockMutexes();
                                                }

                                                else if(client.receive(packet) == sf::Socket::Error)
                                                {
                                                        std::cout << "NetworkingError";
                                                }
                                        }
                                }
                        }
                }
        }
}
 

The handlePacket() function takes a packet, unloads it, and does something with the data (puts in some movement requests, tile changes, etc).

The mu::lock/unlockMutexes() functions lock all the global mutexes to prevent deadlock and crashes. Yes, it's a pain to do multithreading. I have to lock everything every time a thread accesses some sort of data within the program. I'm beginning to wonder if multi-threading was even worth it.

The thing I don't understand is why it doesn't just freeze up completely (because of deadlock or failing to unlock a mutex)-- it always resumes normal work receiving packets/handling data/saving after about 10-30 seconds (in some bad cases, longer). I also tested to make sure that it didn't have to do with the loading/saving aspects of the program and it seemed to be normally timed. The thing is that it only freezes like this when there are multiple players online (all of them seem to be doing normal tasks in the game-- nothing out of the ordinary that would execute some long function).

If you want to have an extended look at it, you can add me on Skype (brady.welch).

binary1248

  • SFML Team
  • Hero Member
  • *****
  • Posts: 1405
  • I am awesome.
    • View Profile
    • The server that really shouldn't be running
Re: TCP Lag Reduction Techniques?
« Reply #9 on: April 03, 2013, 12:51:20 am »
if(client.receive(packet) == sf::Socket::Done)
{
        ...
}

else if(client.receive(packet) == sf::Socket::Disconnected)
{
        ...
}

else if(client.receive(packet) == sf::Socket::Error)
{
        ...
}
 
This is bogus. You try to receive multiple times per "selector-ready-status" because you need to evaluate the if statements independently of one another. It means if the first .receive() doesn't return sf::Socket::Done you will try to .receive() again although there might not be data available. The same for the third case. What you want to do is receive once and save the return status to differentiate between the different cases that have to be handled.
sf::Socket::Status status = client.receive(packet);

if(status == sf::Socket::Done)
{
        ...
}

else if(status == sf::Socket::Disconnected)
{
        ...
}

else if(status == sf::Socket::Error)
{
        ...
}
 
SFGUI # SFNUL # GLS # Wyrm <- Why do I waste my time on such a useless project? Because I am awesome (first meaning).

Jungletoe

  • Full Member
  • ***
  • Posts: 132
    • View Profile
    • Email
Re: TCP Lag Reduction Techniques?
« Reply #10 on: April 04, 2013, 10:08:06 pm »
Ok, I changed that, but I'm still getting spikes.

binary1248

  • SFML Team
  • Hero Member
  • *****
  • Posts: 1405
  • I am awesome.
    • View Profile
    • The server that really shouldn't be running
Re: TCP Lag Reduction Techniques?
« Reply #11 on: April 05, 2013, 12:56:23 am »
Well your usage of mutexes in that snippet is also really strange. Do you have multiple recPackets threads? That's not such a good idea since you only protect the Selector with a mutex when a new connection is accepted and when a connection disconnects. When you call wait() on the Selector it is also modified internally so that would also need to be protected with a mutex, but that would make your whole recPackets method single threaded anyway. I don't really see why you would need more than one thread doing the network processing since the majority of it takes place in the sections that you already guard with a mutex. Also try to avoid locking multiple mutexes at the same time. Mutexes are meant to protect against concurrent access to resources. Locking the client list mutex and handling a player disconnect isn't something I would do because there is a higher risk of deadlocks if too much happens in between a lock and unlock. The same applies for handlePacket. Move the locks and unlocks to the resource access sites and it'll be easier to debug deadlocks which I'm sure are happening.

If you ask me, I still say that the easiest way out of this problem is to make your networking code run in the same thread as the main application. All you have to do is get rid of the while(running) and just call recPackets() every frame, you can also lower the wait() time on the selector to some very small value so it won't even influence the framerate of the server. That way you can also get rid of all those complicated mutexes.
SFGUI # SFNUL # GLS # Wyrm <- Why do I waste my time on such a useless project? Because I am awesome (first meaning).

Jungletoe

  • Full Member
  • ***
  • Posts: 132
    • View Profile
    • Email
Re: TCP Lag Reduction Techniques?
« Reply #12 on: April 05, 2013, 04:30:01 am »
Well your usage of mutexes in that snippet is also really strange. Do you have multiple recPackets threads? That's not such a good idea since you only protect the Selector with a mutex when a new connection is accepted and when a connection disconnects.

There is only one recPackets thread. I actually have a global mutex which locks up the whole program every time it receives a packet it needs to handle or needs to execute the worker thread. It's extremely inefficient as it is, and I really need to redo it. Without the global mutex, everything kept getting deadlocked.

binary1248

  • SFML Team
  • Hero Member
  • *****
  • Posts: 1405
  • I am awesome.
    • View Profile
    • The server that really shouldn't be running
Re: TCP Lag Reduction Techniques?
« Reply #13 on: April 05, 2013, 05:18:26 am »
Well, I guess the first thing to do is rework how the program handles packets. Using global mutexes is "unhealthy" and would lead to maintainability problems sooner or later. Even if you didn't have this problem now, the larger your codebase becomes the more of a headache it will be to fix strange problems like this in the future.
SFGUI # SFNUL # GLS # Wyrm <- Why do I waste my time on such a useless project? Because I am awesome (first meaning).

Jungletoe

  • Full Member
  • ***
  • Posts: 132
    • View Profile
    • Email
Re: TCP Lag Reduction Techniques?
« Reply #14 on: April 05, 2013, 06:33:04 am »
What if I have a thread that handles the packets and another one that receives them from the selector? The selector thread would simply dump all of the unprocessed packets into a vector of packets while the handler thread would pick up those packets and do something with them.

That way, it wouldn't get frozen up and delay connections from being acknowledged and stuff.