Non-Blocking TCP Freeze [Fixed]

[Jungletoe]

Client Side Only (runtime - crash)
http://fbe.am/kMT
http://fbe.am/kMU

Client & Server (after crash)
http://fbe.am/kMW

[binary1248]

Well, from the capture I can only see a few retransmits which is pretty normal for a connection over the internet. Data isn't really being lost at a lower layer as far as I can tell from the capture.

As for the 5 minute reconnect... When TCP shuts down the stream because of some communication error it normally transits into the TIME_WAIT state skipping the CLOSE_WAIT on the remote end. TIME_WAIT is a timeout that is required according to the TCP specification to allow any data still in transit to reach its destination before the endpoints on the hosts really disappear (transit into the CLOSED state). For certain reasons this timeout was chosen to be 2 * Maximum Segment Lifetime which is 2 * 2 minutes. As such on any communication failure you cannot reuse the same port numbers for both sides as this port number pair along with source and destination address uniquely identify a TCP connection. Why does your client try to reestablish a new connection with the same source port? I don't know

If you are sure that you will be the only one using the port number you could try to enable the SO_REUSEADDR as described here: http://en.sfml-dev.org/forums/index.php?topic=11545.0 and see if it makes any difference. Other than that I have no clue what could be the reason for the 5 minutes. Maybe you can send me a copy of the client executable so I can try it for myself?

[Jungletoe]

I sent you the client via PM.

By the way, people from the UK are having a hard time connecting. It always shows offline for them but for nobody else. I already got rid of the connection timeout. It just seems odd that Brits can't connect consistently, yet Russians and Japanese people have no problems whatsoever (the server is located in Eastern US).

[binary1248]

This is common . I live in Germany and can't connect at all to some popular game servers sometimes. This is due to routing "failures" or just a "grudge" one ISP has against another. Most of the time they are temporary but sometimes they are more long-lived. The only thing you can do in that case is either colocate your server resources, a.k.a. have a server in Europe as well, or establish a sort of gateway/proxy/tunnel server in Europe that forwards connections to the server in the US. Other than that, you can resort to the good will of the engineers of the affected ISPs and hope they will resolve the problem with the information you provide them .

[Jungletoe]

Ah ok.

So did you have a chance to download the client?

[binary1248]

I will do so when I have a bigger chunk of time on my hands .

[Jungletoe]

I will do so when I have a bigger chunk of time on my hands .

Ok gotcha.

[Jungletoe]

Ok, so it's almost been a week. Not to be impatient, but have you had any time to check it out?

[Jungletoe]

Screw it. I ran my own tests. It seems that it's a problem with my object packets being too big.

I ran a test where I sent 20 very large packet (20,000 bytes) and the client suddenly stopped receiving packets and corrupted/dropped most of the packets (except 1).

Next, I sent 20,000 packets with 20 bytes. No problems.

Overall, the same amount of data was sent, but bigger packets seem to corrupt and get trashed. Wireshark picked up that the packets were still being sent around, just the rec() funct wasn't working correctly. I believe this is a bug with SFML.

Why does this happen? Beats me, but I'd like an answer and a possible solution. Thanks.

[Jungletoe]

...and I can't find a fix. Can you guys provide some support on this? It's obviously a large flaw that's been ignored for quite some time.

I've been asking for support for 2 months on this issue and have gotten nowhere. My friend who's a fan of BSD sockets has been telling me to switch away from SFML for some time now, and I'm starting to trust him more and more. I'm extremely pissed off that all of the work I've put into the backend for my whole game is now defunct, so I would really like an immediate explanation. Months of putting up with complaints of my community is starting to take a toll on me. This is embarrassing for both SFML and harmful to me financially now that players have left the game due to the freezes.

[binary1248]

Yeah, I had a lot of university work to do over the last week so I wasn't able to do any SFML related work. I tried to download the client however it just gave me a 404 error?

Oh and in case you are completely restless, I can tell you it is unwise switching to BSD sockets until you have determined where the exact problem really is. In the worst case you might end up having the same issue even when using BSD sockets and wasted a ton of time needlessly rewriting a lot of code.

Just on a side note, how much bandwidth does the client require to function properly, as in be played without too many strange things happening?

[Jungletoe]

Yeah, I had a lot of university work to do over the last week so I wasn't able to do any SFML related work.

Ah sorry. I've been stressed out as well between this problem and exams. Sorry if I sound annoying or pushy, it's just that I'm really mad at this whole situation. My anxiety is catching up with me

I tried to download the client however it just gave me a 404 error?

Sorry, use http://colonies-game.com/downloads.php

Oh and in case you are completely restless, I can tell you it is unwise switching to BSD sockets until you have determined where the exact problem really is. In the worst case you might end up having the same issue even when using BSD sockets and wasted a ton of time needlessly rewriting a lot of code.

That's my fear. My constant problem is just lack of control in this situation though, which I guess is what happens when I chose a highlevel library.

Just on a side note, how much bandwidth does the client require to function properly, as in be played without too many strange things happening?

How would I test this (I'm nearly 100% certain it has to do with bandwidth now)?

EDIT:
Ran some more tests. Apparently it only starts screwing up if you send multiple huge packets, not just one.

EDIT 2:
It appears that the max size of a packet I can send repeatedly (more than 2 times) without corrupting/stop receiving/dropping is 65472 bytes. If I make the packets smaller, they won't error out, but will error out after being sent more times than 5.

[binary1248]

I tried running the client. I couldn't notice any strange behaviour regarding the traffic, then again all the packets being sent and received were smaller than 200 bytes.

Is there a way to make the client send/receive larger packets? Maybe something you would have to do in game...

Also... is there a reason why you need to send such large packets over the network? Generally sending such large data chunks at once and letting lower layers take care of fragmentation is an unhealthy practice...

[Jungletoe]

I tried running the client. I couldn't notice any strange behaviour regarding the traffic, then again all the packets being sent and received were smaller than 200 bytes.

Is there a way to make the client send/receive larger packets? Maybe something you would have to do in game...

I recently broke down the packet sizes a bit. Freezes usually happen when people are walking and loading new chunks, or loading the game. They seem to happen a lot more frequently when loads of people are on at once.

Try relogging multiple times in the forest right next to spawn. Spamming chat messages also increases the frequency (if you need someone to spam chat, let me know).

Also... is there a reason why you need to send such large packets over the network? Generally sending such large data chunks at once and letting lower layers take care of fragmentation is an unhealthy practice...

Map data chunks. Basically, there are 20x20 chunks which each hold a tileID (Int32) and a transitionID (Int32)... and those are just the map chunks. It also has to send animal chunks, object chunks, and dropped object chunks.

Before I broke down the object chunks and made it send each individual object alone (instead of large chunks), it would often start distorting around object chunk sending. Now it seems that dropped item chunks and animals seem to crash the client a lot more.

I'm working on replicating the bug through a minimal and complete example at the moment.

[Jungletoe]

Ok, here is some code that will replicate it.

Send this from a server to a client:

if(recievedPacket) // Treat the client-sent packet as a request for the server to respond
        {
                for(sf::Int32 j = 0; j < 5; j++)
                {
                        sf::Packet* spack = new sf::Packet;
                        sf::Int32 packID = 99;
                        *spack << packID;

                        for(sf::Int32 x = 0; x < 2050; x++)
                        {
                                sf::Int32 num = 11;
                                *spack << num;
                        }

                        socket.send(*spack);
                }
        }
 

As you can see, it sends that large packet 5 times. It will work the first 2 times. The third and fourth time, it will be distorted. The fifth time, it will be dropped (along with any packets of any size sent afterwards-- thus the recv() freeze).