Welcome, Guest. Please login or register. Did you miss your activation email?

Author Topic: Non-Blocking TCP Crashing Client?  (Read 6106 times)

0 Members and 2 Guests are viewing this topic.

Jungletoe

  • Full Member
  • ***
  • Posts: 132
    • View Profile
    • Email
Non-Blocking TCP Crashing Client?
« on: April 20, 2013, 02:58:59 am »
Overview
Ok, sorry to bug you guys and make a new thread, but I feel that the other one was titled wrongly for what I' now encountering.

The server is now stable thanks to Binary's help, but my client is very unstable. Again, I cannot replicate any of the crashes myself, which leads me to believe that people who live further away (with greater ping times) are corrupting the packets somehow. I don't know how this is possible, but it's crippling the stability of my playerbase and I have absolutely no way of testing it.

Details:
I am using unblocking TCP sockets, although I switch to blocking when I connect() from the client.

Code

Server send code:
void NetworkManager::sendPacket(sf::Packet* pack, sf::Int32 connectorID, bool delPacket)
{
        if(getConnectorWithID(connectorID) != NULL)
        {
                Connector* c = getConnectorWithID(connectorID);
                if(c != NULL)
                        c->sendPacket(pack);
        }

        if(delPacket)
        {
                //sf::Int32 packID;
                //*pack >> packID;
                //std::cout << "Sending :: " << packID << "\n";

                delete pack;
        }
}
 


Retry send if it fails or returns sf::Socket::NotReady:
void NetworkManager::retryDroppedPackets()
{
        for(sf::Int32 i = 0; i < ent::connectors.size(); i++)
        {
                if(ent::connectors[i]->packets.size() > 0)
                {
                        sf::Socket::Status status = ent::connectors[i]->socket.send(ent::connectors[i]->packets.front());

                        if(status != sf::Socket::NotReady && status != sf::Socket::Error)
                        {
                                ent::connectors[i]->packets.pop();

                                if(status == sf::Socket::Error)
                                {
                                        std::cout << "E1E\n";
                                }
                        }
                }
        }
}
 


Server receive code:
void NetworkManager::recPackets()
{
        Connector* newConnector = new Connector;
        newConnector->socket.setBlocking(false);
        newConnector->player = NULL;

        if(listener.accept(newConnector->socket) == sf::Socket::Done)
        {
                //      Adds a new client to client list
                maxConID += 1;
                newConnector->setID(maxConID);
                ent::connectors.push_back(newConnector);
        }

        else
        {
                //      Nothing we can do...
                delete newConnector;
        }


        //      The listener socket is not ready, test all other sockets
        for(sf::Int32 i = 0; i < ent::connectors.size(); i++)
        {
                sf::Packet packet;
                sf::Socket::Status status = ent::connectors[i]->socket.receive(packet);

                if(status == sf::Socket::Done)
                {
                        ent::connectors[i]->logClock.restart();

                        sf::Uint32 packetCode;
                        packet >> packetCode;

                        handlePacket(packetCode, packet, ent::connectors[i]->getID());
                }

                else if(status == sf::Socket::Disconnected)
                {
                        ent::playerHandler.handleDisconnect(ent::connectors[i]->getID());
                }

                else if(status == sf::Socket::Error)
                {
                        std::cout << "NetworkingError";
                }

                else if(status == sf::Socket::NotReady)
                {
                        //      Not ready
                        //std::cout << "not ready";
                }
        }
}
 


Client send code:
void NetworkManager::sendPacket(sf::Packet* pack)
{
        socket.setBlocking(false);
        if(droppedPackets.size() <= 0)
        {
                sf::Socket::Status status = socket.send(*pack);

                if(status == sf::Socket::Done)
                {
                        delete pack;
                }

                else if(status == sf::Socket::NotReady)
                {
                        droppedPackets.push(pack);
                }

                else if(status == sf::Socket::Error)
                {
                        std::cout << "ERROR";
                        droppedPackets.push(pack);
                }

                else if(status == sf::Socket::Disconnected)
                {
                        if(ent::game.isRunning())
                        {
                                ent::game.disconnect();
                                delete pack;
                        }
                }
        }

        else
        {
                droppedPackets.push(pack);
        }
}
 


Server send code if first time fails:
void NetworkManager::retryPackets()
{
        if(droppedPackets.size() > 0)
        {
                socket.setBlocking(false);
                sf::Socket::Status status = socket.send(*droppedPackets.front());

                if(status == sf::Socket::Done)
                {
                        delete droppedPackets.front();
                        droppedPackets.pop();
                }

                else if(status == sf::Socket::NotReady)
                {
                        //      Do nothing

                }

                else if(status == sf::Socket::Error)
                {
                        std::cout << "ERROR";
                        delete droppedPackets.front();
                        droppedPackets.pop();
                }

                else if(status == sf::Socket::Disconnected)
                {
                        if(ent::game.isRunning())
                        {
                                ent::game.disconnect();
                                delete droppedPackets.front();
                                droppedPackets.pop();
                        }
                }
        }
}
 


Client receive:
void NetworkManager::handle()
{
        retryPackets();

        socket.setBlocking(false);

        sf::Packet pack;
        sf::Socket::Status status = socket.receive(pack);

        if(status == sf::Socket::Done)
        {
                sf::Int32 packetType;           //      Determines what the packet contains, different packet types contain different info

                if(pack >> packetType)
                {
                        if(displayPacketID)
                                std::cout << "\nPacket Type :: " << packetType;
//
//
//                    Handle Packet
//
//

}

                else
                {
                        if(displayPacketID)
                                std::cout << "Junk Packet";
                }
        }

        else
        {
                //      not ready
        }
}
 


Client connect code (runs on its own thread):
bool NetworkManager::connect(std::string IP, sf::Int32 port)
{
        if(ip != NULL)
                delete ip;

        ip = new sf::IpAddress(IP);

        socket.setBlocking(true);
        sf::Socket::Status status = socket.connect(*ip, port, sf::milliseconds(250));
        socket.setBlocking(false);

        if(status == sf::Socket::Disconnected)
        {
                //std::cout << "disconnected";
                connected = false;
                gl::Vars::connected = false;
                return false;
        }

        else if(status == sf::Socket::Done)
        {

                //std::cout << "connected";
                gl::Vars::connected = true;
                connected = true;

                sf::Packet* spack = new sf::Packet;
                sf::Int32 packID = 0;
                *spack << packID << gl::Vars::versionNumber;
                ent::networkManager.sendPacket(spack);

                return true;
        }

        else if(status == sf::Socket::Error)
        {
                connected = false;
                gl::Vars::connected = false;

                return false;
        }

        else
        {
                std::cout << "not ready";
                connected = false;
                gl::Vars::connected = false;

                return false;
        }
}


Client Main:
ent::menu.Run();

        if(ent::menu.getSuccess())      //      If logged in
        {
                ent::loadingScreen.Run();
                if(ent::game.Run())             //      if dead
                {
                        ent::death.Run();
                }
        }


Game.Run()
Init();
        cleanClock.restart();
        while(running && !dead)
        {
                ent::networkManager.handle();
                Input();
                Data();
                Clear();
                Draw();
                if(cleanClock.getElapsedTime().asSeconds() >= 10.0f)
                {
                        cleanClock.restart();
                        Clean();
                }
        }
        cleanUp();

        return dead;


I'd love if one of you veterans could add me on Skype (brady.welch) so we can resolve this issue. I could send you the full source and client for you to test for my game.
« Last Edit: April 20, 2013, 03:01:20 am by Jungletoe »

binary1248

  • SFML Team
  • Hero Member
  • *****
  • Posts: 1405
  • I am awesome.
    • View Profile
    • The server that really shouldn't be running
Re: Non-Blocking TCP Crashing Client?
« Reply #1 on: April 20, 2013, 04:49:25 pm »
Why don't you handle the other return codes from the client .receive()? You only check to see if the receive was successful and don't differentiate the other cases. The connection might have been disconnected but unless you removed the handling code for compactness of the post, you aren't dealing with the cases where TCP determines it can't guarantee reliable transmission and shuts the connection down by itself without your consent. As a rule of thumb, you must handle all return codes, everywhere. It makes debugging much easier.

I don't know how versed your test users are, but if it is possible, you could ask those that are experiencing these problems to send you a Wireshark trace or similar packet dump of the data they are receiving and sending. You can then compare this to the dump of the server and see if packets are indeed being lost in transmission. If this is not the case then you know it is something depending on the application only and you would look in certain places. If packets are lost, then TCP should either try to recover or close the connection appropriately. If you handle this in your code and yet you get corrupted data or the connection is not closed, then it is either a bug in the underlying library (SFML) or operating system network stack (this is very very very unlikely).

If your test users aren't that capable, you can always simulate their kind of connection quality locally. I did a quick Google search and found this: http://stackoverflow.com/questions/614795/simulate-delayed-and-dropped-packets-on-linux

I don't know how familiar you are with Linux, but I don't think Windows offers such functionality built into their networking subsystem. You can always set up a virtual machine running Linux and test the server from there. This method does not work with the client since you can only modify the characteristics of the outgoing traffic.
SFGUI # SFNUL # GLS # Wyrm <- Why do I waste my time on such a useless project? Because I am awesome (first meaning).

Jungletoe

  • Full Member
  • ***
  • Posts: 132
    • View Profile
    • Email
Re: Non-Blocking TCP Crashing Client?
« Reply #2 on: April 20, 2013, 08:32:45 pm »
Ok, I'll try some of those methods later.

Apparently, the crashes are actually freezes. I had one of my testers use Very Sleepy to track the processes of the threads and this is what he got:


This is the exact same info and process that it was stuck on with the server before I switched to non-blocking. Oddly enough, non-blocking on the client doesn't do anything about it. I've narrowed the whole program down to two threads (connecting and main) and that's what always happens. It may also have something to do with sounds, although I doubt it.

By the way, this has been happening ever since I switched the server networking to non-blocking, which seems very odd.
« Last Edit: April 20, 2013, 08:35:13 pm by Jungletoe »

binary1248

  • SFML Team
  • Hero Member
  • *****
  • Posts: 1405
  • I am awesome.
    • View Profile
    • The server that really shouldn't be running
Re: Non-Blocking TCP Crashing Client?
« Reply #3 on: April 20, 2013, 09:02:35 pm »
By the way, this has been happening ever since I switched the server networking to non-blocking, which seems very odd.
Well, since you changed the server to non-blocking you know that it does what you expect it to do. It might change the order or the rate at which events are handled which might have revealed bugs in the client which remained hidden until then. But be happy you discovered them so soon in development. These bugs are often ticking time bombs that are a pain later on in the development of software.

The WaitForMultipleObjects entry in Sleepy also hints at the usage of a Mutex lock that might be causing the freezing in the client. So maybe you might want to investigate those and see which one locks indefinitely, for example by outputing all the locks and unlocks to a log.txt for debug purposes and having your testers send those to you.
SFGUI # SFNUL # GLS # Wyrm <- Why do I waste my time on such a useless project? Because I am awesome (first meaning).

Jungletoe

  • Full Member
  • ***
  • Posts: 132
    • View Profile
    • Email
Re: Non-Blocking TCP Crashing Client?
« Reply #4 on: April 20, 2013, 09:46:18 pm »
The WaitForMultipleObjects entry in Sleepy also hints at the usage of a Mutex lock that might be causing the freezing in the client. So maybe you might want to investigate those and see which one locks indefinitely, for example by outputing all the locks and unlocks to a log.txt for debug purposes and having your testers send those to you.

Well that's the thing-- I'm not using any mutexes at all. All the connection thread does is connect(), so I didn't see a point.

binary1248

  • SFML Team
  • Hero Member
  • *****
  • Posts: 1405
  • I am awesome.
    • View Profile
    • The server that really shouldn't be running
Re: Non-Blocking TCP Crashing Client?
« Reply #5 on: April 20, 2013, 11:28:34 pm »
Well, I guess it's just a Windows internal call then. I don't know how Microsoft implemented its blocking POSIX network API, but it could be a possibility that they use WaitForMultipleObjects to block on what would otherwise be an asynchronous network call. Maybe you can trace back through the calls from WaitForMultipleObjects in a debugger when it gets called and see which functions invoke it. That way you know what to log and see where the client gets stuck.
SFGUI # SFNUL # GLS # Wyrm <- Why do I waste my time on such a useless project? Because I am awesome (first meaning).

Jungletoe

  • Full Member
  • ***
  • Posts: 132
    • View Profile
    • Email
Re: Non-Blocking TCP Crashing Client?
« Reply #6 on: April 21, 2013, 12:06:48 am »
Ok, while I'm waiting for that error to reproduce, do you think it has to do with sounds and audio? This error randomly happens as well:


This is all leading me to believe that sounds have something to do with it. I'm pretty sure sounds use multiple threads, so that would explain the freezes. This error happens randomly, even to me.

One of my testers also reported this:


If you look over on the callstack (right hand side), it appears that there were some other SFML functions being called, but without any of my functions. This leads me to believe that it has to do with sounds even more.

Do sounds operate on different threads than the main loop? I assume so.

Edit:
I have confirmed that it has to do with sounds. Closing this thread and starting a new one in the sound board. Thanks.

Thread is here:
http://en.sfml-dev.org/forums/index.php?topic=11235.msg77784#msg77784
« Last Edit: April 22, 2013, 03:24:42 am by Jungletoe »