-
Ok, so I've been posting my problems on this (http://en.sfml-dev.org/forums/index.php?topic=11221.msg77700#msg77700) thread, which I previously believed were due to networking errors. After further investigation, I realized that the problem is rooted in the sound devices.
I've reinstalled openAL using the libraries provided with SFML, so it's not that.
Here is what Very Sleepy presents me:
(http://puu.sh/2EBVi.png)
If you look at the right hand side on the callstack, it appears that it freezes while playing sounds Here is another snapshot I took:
(http://i.imgur.com/5lceC2e.png)
With this callstack freeze, it appears that it is trying to create a sound instead. Very odd.
Here is my SoundHandler.hpp
#ifndef __SOUNDHANDLER_HPP__
#define __SOUNDHANDLER_HPP__
#include "SFML/System.hpp"
#include "SFML/Audio.hpp"
#include <vector>
class SoundHandler
{
private:
sf::Clock timer;
//////////////////////////
// Sounds //
//////////////////////////
sf::SoundBuffer clickBuffer;
sf::Sound clickSFX;
sf::SoundBuffer chopBuffer;
sf::Sound chopSFX;
sf::SoundBuffer leavesBuffer;
sf::Sound leavesSFX;
sf::SoundBuffer windBuffer;
sf::Sound windSFX;
sf::SoundBuffer stormBuffer;
sf::Sound stormSFX;
sf::SoundBuffer rockBuffer;
sf::Sound rockSFX;
sf::Music deathSFX;
sf::SoundBuffer blood1Buffer;
sf::Sound blood1SFX;
sf::Music birdSFX;
sf::SoundBuffer killBuffer;
sf::Sound killSFX;
sf::SoundBuffer popBuffer;
sf::Sound popSFX;
sf::SoundBuffer splatBuffer;
sf::Sound splatSFX;
sf::SoundBuffer chargeBuffer;
sf::Sound chargeSFX;
sf::SoundBuffer splashBuffer;
sf::Sound splashSFX;
sf::SoundBuffer waterBuffer;
sf::Sound waterSFX;
sf::SoundBuffer fireBuffer;
sf::Sound fireSFX;
sf::Music cricketsSFX;
sf::Music musicboxSFX;
sf::Music combatSFX;
sf::SoundBuffer typeBuffer;
sf::Sound typeSFX;
sf::Music feetSFX;
sf::SoundBuffer puffBuffer;
sf::Sound puffSFX;
sf::SoundBuffer stabBuffer;
sf::Sound stabSFX;
//////////////////////////
// Music //
//////////////////////////
std::vector<sf::Int32> playlist;
sf::Int32 playNumber;
sf::Music ambient1;
sf::Music ambient2;
sf::Music ambient3;
sf::Music ourHome;
sf::Music fishing;
sf::Music intro;
sf::Music night;
sf::Music main;
bool firstPlayed;
// Other vars
bool soundMuted;
bool musicMuted;
public:
SoundHandler();
void load();
void handleMusic();
void stopMusic();
bool musicOn;
void muteMusic();
void muteSound();
bool playClick();
void stopClick();
bool playWind();
void stopWind();
bool playChop();
void stopChop();
bool playLeaves();
void stopLeaves();
bool playStorm();
void stopStorm();
bool playRock();
void stopRock();
bool playDeath();
void stopDeath();
bool playBlood1();
void stopBlood1();
bool playBirds();
void stopBirds();
bool playKill();
void stopKill();
bool playPop();
void stopPop();
bool playSplat();
void stopSplat();
void playCharge();
void resetCharge();
void playFishing();
void stopFishing();
void playSplash();
void playWater();
void playCrickets();
void stopCrickets();
void playMain();
void playNight();
void playFire();
void playMusicbox();
void stopMusicbox();
void playCombat();
void playType();
void playFeet();
void playPuff();
void playStab();
};
#endif
...and my SoundHandler.cpp
#include "stdafx.h"
#include <random>
#include <time.h>
#include "SoundHandler.hpp"
#include "Globals.hpp"
SoundHandler::SoundHandler()
{
load();
musicMuted = false;
soundMuted = false;
}
void SoundHandler::load()
{
//////////////////////////
// Sounds //
//////////////////////////
clickBuffer.loadFromFile("rsc/SFX/click.ogg");
clickSFX.setBuffer(clickBuffer);
chopBuffer.loadFromFile("rsc/SFX/chop.ogg");
chopSFX.setBuffer(chopBuffer);
leavesBuffer.loadFromFile("rsc/SFX/leaves.ogg");
leavesSFX.setBuffer(leavesBuffer);
windBuffer.loadFromFile("rsc/SFX/wind.ogg");
windSFX.setBuffer(windBuffer);
windSFX.setLoop(true);
stormBuffer.loadFromFile("rsc/SFX/storm.ogg");
stormSFX.setBuffer(stormBuffer);
rockBuffer.loadFromFile("rsc/SFX/rock.ogg");
rockSFX.setBuffer(rockBuffer);
deathSFX.openFromFile("rsc/SFX/death.ogg");
blood1Buffer.loadFromFile("rsc/SFX/blood1.ogg");
blood1SFX.setBuffer(blood1Buffer);
birdSFX.openFromFile("rsc/SFX/birds.ogg");
killBuffer.loadFromFile("rsc/SFX/kill.ogg");
killSFX.setBuffer(killBuffer);
popBuffer.loadFromFile("rsc/SFX/pop.ogg");
popSFX.setBuffer(popBuffer);
splatBuffer.loadFromFile("rsc/SFX/splat.ogg");
splatSFX.setBuffer(splatBuffer);
chargeBuffer.loadFromFile("rsc/SFX/charge.ogg");
chargeSFX.setBuffer(chargeBuffer);
resetCharge();
splashBuffer.loadFromFile("rsc/SFX/splash.ogg");
splashSFX.setBuffer(splashBuffer);
waterBuffer.loadFromFile("rsc/SFX/water.ogg");
waterSFX.setBuffer(waterBuffer);
waterSFX.setVolume(25);
cricketsSFX.openFromFile("rsc/SFX/crickets.ogg");
fireBuffer.loadFromFile("rsc/SFX/fire.ogg");
fireSFX.setBuffer(fireBuffer);
musicboxSFX.openFromFile("rsc/SFX/musicbox.ogg");
musicboxSFX.setLoop(true);
combatSFX.openFromFile("rsc/SFX/combat.ogg");
typeBuffer.loadFromFile("rsc/SFX/type.ogg");
typeSFX.setBuffer(typeBuffer);
feetSFX.openFromFile("rsc/SFX/feet.ogg");
puffBuffer.loadFromFile("rsc/SFX/puff.ogg");
puffSFX.setBuffer(puffBuffer);
stabBuffer.loadFromFile("rsc/SFX/stab.ogg");
stabSFX.setBuffer(stabBuffer);
//////////////////////////
// Music //
//////////////////////////
// Day Playlist
ourHome.openFromFile("rsc/Music/ourHome.ogg"); // 0
playlist.push_back(0);
ambient1.openFromFile("rsc/Music/ambience1.ogg"); // 1
playlist.push_back(1);
ambient2.openFromFile("rsc/Music/ambience2.ogg"); // 2
playlist.push_back(2);
night.openFromFile("rsc/Music/night.ogg"); // 3
playlist.push_back(3);
ambient3.openFromFile("rsc/Music/ambience3.ogg"); // 4
playlist.push_back(4);
musicOn = true;
firstPlayed = false;
srand(time(NULL));
std::random_shuffle(playlist.begin(), playlist.end());
playNumber = 0;
// Fishing
fishing.openFromFile("rsc/Music/fishing.ogg");
fishing.setLoop(true);
// Intro
intro.openFromFile("rsc/Music/intro.ogg");
intro.setLoop(true);
// Main
main.openFromFile("rsc/Music/The Colony.ogg");
main.setLoop(true);
}
void SoundHandler::handleMusic()
{
if(((timer.getElapsedTime().asSeconds() >= 300) || ((!firstPlayed) && (timer.getElapsedTime().asSeconds() >= 60))) && musicOn && !musicMuted)
{
firstPlayed = true;
sf::Int32 songNumber = playlist[playNumber];
bool restartTimer = true;
if(songNumber == 0)
{
ourHome.play();
}
else if(songNumber == 1)
{
ambient1.play();
}
else if(songNumber == 2)
{
ambient2.play();
}
else if((songNumber == 3))
{
if((gl::Vars::timeOfDay == NIGHT) || (gl::Vars::timeOfDay == DAWN) || (gl::Vars::timeOfDay == DAWN))
night.play();
else
restartTimer = false;
}
else if((songNumber == 4))
{
ambient3.play();
}
if(restartTimer)
timer.restart();
playNumber++;
if(playNumber >= playlist.size())
{
srand(time(NULL));
std::random_shuffle(playlist.begin(), playlist.end());
playNumber = 0;
}
}
else if(timer.getElapsedTime().asSeconds() >= 300)
{
timer.restart();
}
}
void SoundHandler::stopMusic()
{
ourHome.stop();
ambient1.stop();
ambient2.stop();
ambient3.stop();
fishing.stop();
intro.stop();
night.stop();
musicboxSFX.stop();
main.stop();
}
void SoundHandler::muteMusic()
{
musicMuted = !musicMuted;
if(!musicMuted)
stopMusic();
}
void SoundHandler::muteSound()
{
soundMuted = !soundMuted;
windSFX.stop();
feetSFX.stop();
cricketsSFX.stop();
birdSFX.stop();
}
bool SoundHandler::playChop()
{
if(!soundMuted)
chopSFX.play();
return true;
}
void SoundHandler::stopChop()
{
chopSFX.stop();
}
bool SoundHandler::playClick()
{
if(!soundMuted)
clickSFX.play();
return true;
}
void SoundHandler::stopClick()
{
clickSFX.stop();
}
bool SoundHandler::playLeaves()
{
if(!soundMuted)
leavesSFX.play();
return true;
}
void SoundHandler::stopLeaves()
{
leavesSFX.stop();
}
bool SoundHandler::playWind()
{
if(!soundMuted)
windSFX.play();
return true;
}
void SoundHandler::stopWind()
{
windSFX.stop();
}
bool SoundHandler::playStorm()
{
if(!soundMuted)
stormSFX.play();
return true;
}
void SoundHandler::stopStorm()
{
stormSFX.stop();
}
bool SoundHandler::playRock()
{
if(!soundMuted)
rockSFX.play();
return true;
}
void SoundHandler::stopRock()
{
rockSFX.stop();
}
bool SoundHandler::playDeath()
{
if(!musicMuted)
deathSFX.play();
return true;
}
void SoundHandler::stopDeath()
{
deathSFX.stop();
}
bool SoundHandler::playBlood1()
{
if(!soundMuted)
blood1SFX.play();
return true;
}
void SoundHandler::stopBlood1()
{
blood1SFX.stop();
}
bool SoundHandler::playBirds()
{
if(!soundMuted)
birdSFX.play();
return true;
}
void SoundHandler::stopBirds()
{
birdSFX.stop();
}
bool SoundHandler::playKill()
{
if(!soundMuted)
killSFX.play();
return true;
}
void SoundHandler::stopKill()
{
killSFX.stop();
}
bool SoundHandler::playPop()
{
if(!soundMuted)
popSFX.play();
return true;
}
void SoundHandler::stopPop()
{
popSFX.stop();
}
bool SoundHandler::playSplat()
{
if(!soundMuted)
splatSFX.play();
return true;
}
void SoundHandler::stopSplat()
{
splatSFX.stop();
}
void SoundHandler::playCharge()
{
if(!soundMuted)
chargeSFX.play();
chargeSFX.setPitch(chargeSFX.getPitch() + .1);
}
void SoundHandler::resetCharge()
{
chargeSFX.setPitch(.1);
}
void SoundHandler::playFishing()
{
if(!musicMuted)
fishing.play();
}
void SoundHandler::stopFishing()
{
fishing.stop();
}
void SoundHandler::playSplash()
{
if(!soundMuted)
splashSFX.play();
}
void SoundHandler::playWater()
{
waterSFX.setPitch((rand() % 20) * .1 + .1);
if(!soundMuted)
waterSFX.play();
}
void SoundHandler::playCrickets()
{
if(!soundMuted)
cricketsSFX.play();
}
void SoundHandler::stopCrickets()
{
if(!soundMuted)
cricketsSFX.stop();
}
void SoundHandler::playMain()
{
if(!musicMuted)
main.play();
}
void SoundHandler::playNight()
{
if(!soundMuted)
night.play();
}
void SoundHandler::playFire()
{
if(!soundMuted)
fireSFX.play();
}
void SoundHandler::playMusicbox()
{
musicOn = false;
stopMusic();
musicboxSFX.stop();
if(!soundMuted && !musicMuted)
{
musicboxSFX.play();
stabSFX.play();
}
}
void SoundHandler::stopMusicbox()
{
musicboxSFX.stop();
}
void SoundHandler::playCombat()
{
if(!soundMuted)
combatSFX.play();
}
void SoundHandler::playType()
{
if(!soundMuted)
typeSFX.play();
}
void SoundHandler::playFeet()
{
if(!soundMuted)
feetSFX.play();
}
void SoundHandler::playPuff()
{
if(!soundMuted)
puffSFX.play();
}
void SoundHandler::playStab()
{
if(!soundMuted)
stabSFX.play();
}
For the sake of "minimal and complete examples", here is an example main function:
#include "SoundHandler.hpp"
int main()
{
SoundHandler soundHandler;
soundHandler.load();
soundHandler.handleMusic();
soundHandler.playCombat();
return 0;
}
Of course, this probably will not crash it since it crashes randomly when I play sounds. I haven't been able to solidly reproduce it yet repeatedly, so I apologize for lack of information.
My guess is that it may have to do with music being loaded at the same time as sounds? That's just a guess from my debugging, but I can't be sure.
All of the sounds are <1 second and the music is anywhere from 30 secs to 5 minutes.
Also, this occasionally gets output in my console and in other's:
(http://puu.sh/2CYMN.png)
Any ideas?
-
no idea but you should consider using std::map or alike to reduce your code duplication and increase its flexibility ! ;)
-
std::random_shuffle(playlist.begin(), playlist.end());
This is a very critical line of code. You essentially copy/move the sf::Music objects around in order to shuffle them. sf::Music contains an sf::Mutex and holds a pointer to sf::SoundFile which are both NonCopyable. I have no idea how your compiler copies objects which contain NonCopyable objects but if you ask me this shouldn't be allowed. If you are compiling with C++11 then std::random_shuffle might invoke the default move constructors for those classes which might or might not do what is required to ensure expected behaviour.
You should try storing the sounds in a std::map as Hiura already recommended. That way you would only have to shuffle the keys to your Sounds/Music instead of the objects themselves. Maybe that will solve the problem you are experiencing.
On a side note. Laurent has to make sure that objects that shouldn't be moved if C++11 is enabled also get something like sf::NonMoveable. I don't want to know what happens when you move an object while it is being accessed in another thread :-\.
-
This is a very critical line of code. You essentially copy/move the sf::Music objects around in order to shuffle them.
His playlist variable is a container of integers.
On a side note. Laurent has to make sure that objects that shouldn't be moved if C++11 is enabled also get something like sf::NonMoveable. I don't want to know what happens when you move an object while it is being accessed in another thread
If I remember correctly, the conditions for a compiler-generated move constructor are quite strict. I don't think you get one if a user-defined copy constructor exists.
-
In C++11, the automatic generation of the Big Five* is disabled as soon as at least one of them is defined by the user. One can use =default to explicitly provide them without writing a definition.
____________
* Big Five: Copy constructor, move constructor, copy assignment operator, move assignment operator, destructor
-
I'm not using C++11.
This is a very critical line of code. You essentially copy/move the sf::Music objects around in order to shuffle them.
His playlist variable is a container of integers.
Umm... so what do I do? What's the exact problem?
I think it may have something to do with exhausting OpenAL resources, but I'm not sure.
-
The problem is... that there is no apparent problem :P. Or at least no easily testable hypothesis. Your code is still too "large" to isolate a single source. Does this problem only arise when you have every single line you posted? Also instead of manually running the application many times over, can't you reproduce it by throwing that main() code into a loop and letting it run until it crashes? Also, you might want to check the return values of your sound loading. It might be that a sound fails to load (completely) and you try to play it resulting in a crash.
The callstack doesn't help much unless you can trace it all the way back to an SFML call. For some reason OpenAL isn't as open as one might think and doesn't make it easy to debug applications making use of it. I don't even want to start about debugging using Windows system calls and multiple threads. That is almost like fortune telling.
-
Yeah... so after doing more digging, I don't think it's a sound problem after all since it's not even freezing apparently.
I finally reproduced the bug. Basically, the screen keep drawing but packets are failing to be received. I posted my recv() function over on the other thread... so this is kind of pissing me off. I don't know what else I can do. No packets with the error status are being reported either...
-
Like I said, capture the packets using Wireshark and show us the dump. If the data isn't arriving then there is nothing to be received. But then TCP should close the connection if it doesn't recover and you should see TCP sending a RST. If the data does arrive but your code doesn't get the data then either there is something wrong with SFML or there is something wrong with your code. If the former is the case, then it should be reproducible with a minimal code example. If you can't reproduce it then you can use the minimal code to figure out what the difference is that is causing the problem.
Save yourself the unneeded trouble and verify whether the data is indeed arriving or not arriving at the client. It isn't that hard to install Wireshark and start capturing packets. Just filter out everything that is not specific to your application and post the dumps somewhere for us to inspect.
-
I filtered and dumped everything here. After I froze, it seemed to not get any more packets on Wireshark except 4 extras which randomly came in (2 were even from UDP, which is odd).
http://www.mediafire.com/?bzjp2u7u3lumcw9
-
This is only a dump of one way traffic. Provide the traffic in the other direction as well. It is hard to understand whether TCP is working correctly if you can only see one direction. Also provide the same for the other host (send and receive) so that you can compare both with each other and discover discrepancies.
-
This is only a dump of one way traffic. Provide the traffic in the other direction as well. It is hard to understand whether TCP is working correctly if you can only see one direction. Also provide the same for the other host (send and receive) so that you can compare both with each other and discover discrepancies.
Is this good?
http://www.mediafire.com/?27jmk41zl779lh6
-
Ok, would it be a problem with non-blocking sends if I delete the packet immediately after the send() function? That's the only thing I can think of that could corrupt it.
-
Actually I had a look at the pcap file and there are many things that deviate from what would be a "normal" TCP session.
- Did you capture multiple game sessions within that dump? If not, why are there multiple TCP connections between the 2 hosts? And why are they established in both directions?
- How do you handle a legitimate disconnect? Is there a "disconnect" packet you send? Or do you just wait for someone to close the connection in order to know that they want to stop playing?
- This is not your fault, but I reviewed the networking code in SFML yet again, and there is an inherent problem with how it closes TCP connections.
Because TCP sockets are not shutdown() before being close()d, in specific cases (such as yours) a FIN request is sent in one direction and immediately after any incoming traffic arrives a RST packet is sent to indicate that the socket was already close()d although the opposite side still has data to send. This is the opposite of what would be called a "graceful" TCP connection shutdown employing a four-way handshake and is often classified as bad network programming or even a bug in the networking code. I don't know if this is made worse by the fact that the sockets are set to non-blocking because close() is supposed to return immediately, but I can imagine some implementations where it does have noticeable side-effects.
This is probably the biggest problem I have found in the SFML networking code up to now and probably the most annoying one to determine if it really affects you.
I am interested to hear what Laurent has to say to this. For now, you need to think of a way for both sides to agree that the connection is going to be disconnected and stop sending data before closing the SFML Socket object. Any connection termination that occurs without this negotiation would have to be considered as an unexpected termination and proper cleanup has to be performed.
It might seem stupid that you need to replicate features that should be taken care of at lower layers, but engineers are also only humans.
Read more here (http://ia700609.us.archive.org/22/items/TheUltimateSo_lingerPageOrWhyIsMyTcpNotReliable/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable.html).
-
I just realized that I didn't call socket->disconnect() when closing the connection-- I simply just deleted the connector and socket. I'll see if this plays out better.
- Did you capture multiple game sessions within that dump? If not, why are there multiple TCP connections between the 2 hosts? And why are they established in both directions?
- How do you handle a legitimate disconnect? Is there a "disconnect" packet you send? Or do you just wait for someone to close the connection in order to know that they want to stop playing?
- This is not your fault, but I reviewed the networking code in SFML yet again, and there is an inherent problem with how it closes TCP connections.
- It may have been possible that someone else logged in. And yes, I logged out a few times during that capture session in order to reproduce the errors.
- It only tests to see if the connection is closed when sending/receiving from the sockets
- Interesting... would that have to do with my failure to add disconnect() by any chance?
-
Internally SFML just calls close() on a socket whether you explicitly call disconnect() or just delete the socket. So in this case there isn't much you can do. Laurent will have to change something in the implementation to make TCP connection shutdowns cleaner. This might be what is causing your problems. There is no way to tell until it is changed because SFML is causing RST packets to be sent by TCP even in legitimate disconnects.
In the mean time you should think of a way to inform both sides that the connection should be shut down without relying on TCP telling you that the connection was closed at the remote end.
And regarding my first question: Why does the server establish a connection to the client as well? There are what looks like 2 simultaneous TCP connections between the 2 hosts in the dump. Both sides connect to the other using their own connection. That is what I don't understand. Wouldn't a single connection suffice?
-
And regarding my first question: Why does the server establish a connection to the client as well? There are what looks like 2 simultaneous TCP connections between the 2 hosts in the dump. Both sides connect to the other using their own connection. That is what I don't understand. Wouldn't a single connection suffice?
Umm... I didn't connect both sides...
You can take a look at my networking code again (http://en.sfml-dev.org/forums/index.php?topic=11221.msg77667#msg77667). That's all it does.
-
Well... according to the dump, 192.168.1.14 sent a SYN from port 50282 to 173.73.23.101 port 41011 first. Then 173.73.23.101 sent a SYN from port 1024 back to 192.168.1.14 port 41011. This means that 2 connections are established between the hosts. If only the server is listening for new connections then either the second connection attempt was made by some other application, or something is misbehaving. You can have a look yourself. It's the first 2 packets in the dump.
-
Well... according to the dump, 192.168.1.14 sent a SYN from port 50282 to 173.73.23.101 port 41011 first. Then 173.73.23.101 sent a SYN from port 1024 back to 192.168.1.14 port 41011. This means that 2 connections are established between the hosts. If only the server is listening for new connections then either the second connection attempt was made by some other application, or something is misbehaving. You can have a look yourself. It's the first 2 packets in the dump.
I still have no idea what could've connected it... the server never connects and no outside applications use that port.
-
If you say you are experiencing data loss issues, you should try to replicate it with a simple test program (<100 lines, try sending random numbers with the same seed at one end and verifying them at the other) that transfers a lot of data. See if you can make it lose something the same way you lose data here.
-
If you say you are experiencing data loss issues, you should try to replicate it with a simple test program (<100 lines, try sending random numbers with the same seed at one end and verifying them at the other) that transfers a lot of data. See if you can make it lose something the same way you lose data here.
I mean... that'll take a while but I'll try. I don't exactly have time to experiment with this. My fan-base recently grew to 200 people and they expect features every weekend...
One of the dedicated testers just gave me this from Very Sleepy after a crash:
(http://puu.sh/2OlHG.png)
-
Sleepy indicates that that is Call stack 1 of 5. SFML uses multiple threads while it is running, even more when you make use of multiple audio objects. Most of them will be sleeping the majority of the time and I highly doubt that a sleeping thread can crash an application by itself. It might look like the audio is causing the application to crash but this is a wrong assumption since the audio thread is merely sleeping at the time of the crash. You need to check the other 4 call stacks as well. If one of them indicates that something bad happened in your own code then the problem is probably located there. If you still want to check whether it is the audio that is crashing the application, you can try to switch audio off by means of a switch and let people test it. If it never crashes with the audio off then the crash might very well be because of the audio.
-
http://puu.sh/2QjCL.png
http://puu.sh/2QjDD.png
http://puu.sh/2QjDO.png
http://puu.sh/2QjDZ.png
-
It seems to happen when a lot of high-memory packets are sent at once.
-
Judging from the screenshots of Sleepy you provided, you probably didn't link against debug libraries. The callstacks are more or less useless in helping you find the problem because either a) something got really screwed up and the stack got corrupted or b) starting a new thread doesn't reveal anything above the initiation point. The main thread that runs your application and probably also crashes it is missing and that is the important one.
If you say you have dedicated testers, they should be given proper debug versions of the application and it should be run in a proper debug environment corresponding to your build toolchain. Hooking into an application after or during execution is very unreliable, even more so in Windows than on Linux and more often than not you won't end up with much useful information.
Sleepy also wasn't meant to be used as a generic debugging tool. It is a callgraph profiler as can be seen in the screenshots. As such I wouldn't be surprised if it isn't able to fully recover callstacks after an application has crashed.
It seems to happen when a lot of high-memory packets are sent at once.
Unless you can consistently isolate and reproduce this, it is probably just coincidence or just correlated in some indirect way.
-
Unless you can consistently isolate and reproduce this, it is probably just coincidence or just correlated in some indirect way.
I've confirmed that it has to do with lots of high-memory packets being sent at once.
I guess I can try to replicate it.
-
Okay, I definitely know it has to do with high memory packets. The client basically stops receiving the packets if a lot are sent at once.
Is there any limit on how much memory the network can hold before it's processed in a recieve()?
-
What does the packet capture tell you? Are the packets actually arriving at their destination? Are they even being sent from their source?
-
The packets are being sent from the server to the client but the client is not responding to them because of some distortion or deletion. The problem is with the underlying network, not my code because people have to wait 5 minutes before they can play again-- after closing the client and restarting. The client is not receiving anything since it simply just stops working.
-
Can you show me a packet capture of this happening? From both sides if possible.
-
Client Side Only (runtime - crash)
http://fbe.am/kMT
http://fbe.am/kMU
Client & Server (after crash)
http://fbe.am/kMW
-
Well, from the capture I can only see a few retransmits which is pretty normal for a connection over the internet. Data isn't really being lost at a lower layer as far as I can tell from the capture.
As for the 5 minute reconnect... When TCP shuts down the stream because of some communication error it normally transits into the TIME_WAIT state skipping the CLOSE_WAIT on the remote end. TIME_WAIT is a timeout that is required according to the TCP specification to allow any data still in transit to reach its destination before the endpoints on the hosts really disappear (transit into the CLOSED state). For certain reasons this timeout was chosen to be 2 * Maximum Segment Lifetime which is 2 * 2 minutes. As such on any communication failure you cannot reuse the same port numbers for both sides as this port number pair along with source and destination address uniquely identify a TCP connection. Why does your client try to reestablish a new connection with the same source port? I don't know :)
If you are sure that you will be the only one using the port number you could try to enable the SO_REUSEADDR as described here: http://en.sfml-dev.org/forums/index.php?topic=11545.0 and see if it makes any difference. Other than that I have no clue what could be the reason for the 5 minutes. Maybe you can send me a copy of the client executable so I can try it for myself? :)
-
I sent you the client via PM.
By the way, people from the UK are having a hard time connecting. It always shows offline for them but for nobody else. I already got rid of the connection timeout. It just seems odd that Brits can't connect consistently, yet Russians and Japanese people have no problems whatsoever (the server is located in Eastern US).
-
This is common :). I live in Germany and can't connect at all to some popular game servers sometimes. This is due to routing "failures" or just a "grudge" one ISP has against another. Most of the time they are temporary but sometimes they are more long-lived. The only thing you can do in that case is either colocate your server resources, a.k.a. have a server in Europe as well, or establish a sort of gateway/proxy/tunnel server in Europe that forwards connections to the server in the US. Other than that, you can resort to the good will of the engineers of the affected ISPs and hope they will resolve the problem with the information you provide them :).
-
Ah ok.
So did you have a chance to download the client?
-
I will do so when I have a bigger chunk of time on my hands :).
-
I will do so when I have a bigger chunk of time on my hands :).
Ok gotcha.
-
Ok, so it's almost been a week. Not to be impatient, but have you had any time to check it out?
-
Screw it. I ran my own tests. It seems that it's a problem with my object packets being too big.
I ran a test where I sent 20 very large packet (20,000 bytes) and the client suddenly stopped receiving packets and corrupted/dropped most of the packets (except 1).
Next, I sent 20,000 packets with 20 bytes. No problems.
Overall, the same amount of data was sent, but bigger packets seem to corrupt and get trashed. Wireshark picked up that the packets were still being sent around, just the rec() funct wasn't working correctly. I believe this is a bug with SFML.
Why does this happen? Beats me, but I'd like an answer and a possible solution. Thanks.
-
...and I can't find a fix. Can you guys provide some support on this? It's obviously a large flaw that's been ignored for quite some time.
I've been asking for support for 2 months on this issue and have gotten nowhere. My friend who's a fan of BSD sockets has been telling me to switch away from SFML for some time now, and I'm starting to trust him more and more. I'm extremely pissed off that all of the work I've put into the backend for my whole game is now defunct, so I would really like an immediate explanation. Months of putting up with complaints of my community is starting to take a toll on me. This is embarrassing for both SFML and harmful to me financially now that players have left the game due to the freezes.
-
Yeah, I had a lot of university work to do over the last week so I wasn't able to do any SFML related work. I tried to download the client however it just gave me a 404 error? :-\
Oh and in case you are completely restless, I can tell you it is unwise switching to BSD sockets until you have determined where the exact problem really is. In the worst case you might end up having the same issue even when using BSD sockets and wasted a ton of time needlessly rewriting a lot of code.
Just on a side note, how much bandwidth does the client require to function properly, as in be played without too many strange things happening?
-
Yeah, I had a lot of university work to do over the last week so I wasn't able to do any SFML related work.
Ah sorry. I've been stressed out as well between this problem and exams. Sorry if I sound annoying or pushy, it's just that I'm really mad at this whole situation. My anxiety is catching up with me :(
I tried to download the client however it just gave me a 404 error? :-\
Sorry, use http://colonies-game.com/downloads.php
Oh and in case you are completely restless, I can tell you it is unwise switching to BSD sockets until you have determined where the exact problem really is. In the worst case you might end up having the same issue even when using BSD sockets and wasted a ton of time needlessly rewriting a lot of code.
That's my fear. My constant problem is just lack of control in this situation though, which I guess is what happens when I chose a highlevel library.
Just on a side note, how much bandwidth does the client require to function properly, as in be played without too many strange things happening?
How would I test this (I'm nearly 100% certain it has to do with bandwidth now)?
EDIT:
Ran some more tests. Apparently it only starts screwing up if you send multiple huge packets, not just one.
EDIT 2:
It appears that the max size of a packet I can send repeatedly (more than 2 times) without corrupting/stop receiving/dropping is 65472 bytes. If I make the packets smaller, they won't error out, but will error out after being sent more times than 5.
-
I tried running the client. I couldn't notice any strange behaviour regarding the traffic, then again all the packets being sent and received were smaller than 200 bytes.
Is there a way to make the client send/receive larger packets? Maybe something you would have to do in game...
Also... is there a reason why you need to send such large packets over the network? Generally sending such large data chunks at once and letting lower layers take care of fragmentation is an unhealthy practice...
-
I tried running the client. I couldn't notice any strange behaviour regarding the traffic, then again all the packets being sent and received were smaller than 200 bytes.
Is there a way to make the client send/receive larger packets? Maybe something you would have to do in game...
I recently broke down the packet sizes a bit. Freezes usually happen when people are walking and loading new chunks, or loading the game. They seem to happen a lot more frequently when loads of people are on at once.
Try relogging multiple times in the forest right next to spawn. Spamming chat messages also increases the frequency (if you need someone to spam chat, let me know).
Also... is there a reason why you need to send such large packets over the network? Generally sending such large data chunks at once and letting lower layers take care of fragmentation is an unhealthy practice...
Map data chunks. Basically, there are 20x20 chunks which each hold a tileID (Int32) and a transitionID (Int32)... and those are just the map chunks. It also has to send animal chunks, object chunks, and dropped object chunks.
Before I broke down the object chunks and made it send each individual object alone (instead of large chunks), it would often start distorting around object chunk sending. Now it seems that dropped item chunks and animals seem to crash the client a lot more.
I'm working on replicating the bug through a minimal and complete example at the moment.
-
Ok, here is some code that will replicate it.
Send this from a server to a client:
if(recievedPacket) // Treat the client-sent packet as a request for the server to respond
{
for(sf::Int32 j = 0; j < 5; j++)
{
sf::Packet* spack = new sf::Packet;
sf::Int32 packID = 99;
*spack << packID;
for(sf::Int32 x = 0; x < 2050; x++)
{
sf::Int32 num = 11;
*spack << num;
}
socket.send(*spack);
}
}
As you can see, it sends that large packet 5 times. It will work the first 2 times. The third and fourth time, it will be distorted. The fifth time, it will be dropped (along with any packets of any size sent afterwards-- thus the recv() freeze).
-
you should think about tidying up that code first.. until then im not even going to bother to look at it.
-
you should think about tidying up that code first.. until then im not even going to bother to look at it.
How much more clear can I make it? I commented it and provided a rational on what it does.
-
As I told ou in PM, you should provide a complete and minimal example that reproduces the problem, and only that. Don't try to stay in the context of your original application, rewrite it from scratch.
How are we supposed to test or analyze an incomplete piece of code extracted from a big project that we know nothing about?
-
I can see why this it is difficult to reproduce this using a minimal example. The problem obviously doesn't arise from the way SFML implemented packets or TCP communication but from side effects which are difficult to determine and isolate. I assume Jungletoe is reporting about these problems while running the server software on only a single host and thus can only evaluate a single environment. I will test sending and receiving different packet sizes on multiple hosts at my disposal and see if I can come up with anything.
-
I assume Jungletoe is reporting about these problems while running the server software on only a single host and thus can only evaluate a single environment. I will test sending and receiving different packet sizes on multiple hosts at my disposal and see if I can come up with anything.
Awesome, thanks!
-
I just ran the tests pairwise between 3 servers and my home connection and nothing seems to go wrong. SFML can send and receive packets as large as 65000 bytes or packets around 2000 bytes in quick succession without anything being lost at the upper layers. As expected TCP was choking on the data rate a bit and retransmitting data a lot but none of this is noticeable to an application using SFML.
Here is the server/client code I used:
#include <SFML/Network.hpp>
int main() {
sf::TcpListener listener;
listener.listen( 4422 );
sf::TcpSocket socket;
listener.accept( socket );
sf::Packet packet;
while( !socket.receive( packet ) ) {
socket.send( packet );
}
return 0;
}
#include <cstring>
#include <cstdlib>
#include <iostream>
#include <SFML/Network.hpp>
int main() {
sf::TcpSocket socket;
socket.connect( "12.34.56.78", 4422 );
for( int size = 100; size < 65000; size += 100 ) {
std::cout << "Probing size: " << size << "\n";
sf::Packet send_packet;
sf::Packet recv_packet;
for( int i = 0; i < size; i += sizeof( int ) ) {
int value = rand();
send_packet << value;
}
socket.send( send_packet );
socket.receive( recv_packet );
if( ( send_packet.getDataSize() != recv_packet.getDataSize() ) ||
memcmp( send_packet.getData(), recv_packet.getData(), recv_packet.getDataSize() ) ) {
std::cout << "Data corruption!\n";
return 1;
}
}
return 0;
}
-
Your snippet works perfectly, but this doesn't:
int main ( int argc, char *argv[] )
{
sf::TcpSocket socket;
socket.setBlocking(true);
socket.connect("192.168.1.14", 41013);
socket.setBlocking(false);
for(sf::Int32 j = 0; j < 5; j++)
{
// SEND
{
sf::Packet pack;
for(sf::Int32 x = 0; x < 3000; x++)
{
sf::Int32 num = 1;
pack << num;
}
socket.send(pack);
}
std::cout << "Sent packet #" << j << "\n";
}
for(sf::Int32 j = 0; j < 5; j++)
{
// REC
{
sf::Packet pack;
socket.receive(pack);
for(sf::Int32 x = 0; x < 3000; x++)
{
sf::Int32 num;
pack >> num;
if(num != 1)
std::cout << "corruption";
}
std::cout << "Recieved packet #" << j << "\n";
}
}
system("PAUSE");
return 0;
}
Just use the server from your example and use the above as the client.
Edit
Aaaaaand I just discovered the problem. Switching from blocking to non-blocking after connecting SEVERELY distorts packets. I thought non-blocking connect() doesn't work?
Edit 2
Non-blocking connect() does not work. It returns NotReady.
Edit 3
Related topic: http://en.sfml-dev.org/forums/index.php?topic=7435.0
Edit 4
This doesn't corrupt it. My bad-- one second please.
-
I think it is obvious it won't work? The whole program will finish execution in less than 1ms on a decent system because the sockets don't block and the data will have just left the network adapter... Also if you send that fast, you might even overflow the send buffer because there might be latency between your application and the operating system or hardware. You would have to check the return value for the send as well in that case to determine if the data was properly queued for sending.
-
I think it is obvious it won't work? The whole program will finish execution in less than 1ms on a decent system because the sockets don't block and the data will have just left the network adapter... Also if you send that fast, you might even overflow the send buffer because there might be latency between your application and the operating system or hardware. You would have to check the return value for the send as well in that case to determine if the data was properly queued for sending.
Check the edits. The problem appears to be with the non-blocking connect.
Edit
Never mind, this appears to be the same issue. I misread what you said.
I did check the return value, as I showed here: http://en.sfml-dev.org/forums/index.php?topic=11221.0
Edit 2
So what do you suggest I do? I think I'm overflowing the receive buffer in the client because of all the data I'm sending extremely fast.
-
TCP will never "overflow" the receive buffer with data. When the buffer is full TCP
congestion flow control will kick in and implicitly inform the sender that the destination system can't accept any more data. What that will result in is the sending side filling up as well. Once the sending side is filled up, the send() from your application should fail with an error or NotReady in SFML and indicate you should wait a bit and try again. Trust me, the TCP network stack is built in a very robust way. Even over really really crappy connections TCP will never silently drop data. The only place where data can be lost is in your code if you don't listen to what TCP is trying to tell you. If data loss is unavoidable TCP will at least try to notify you of it by closing the connection. In this case, which is also the only case where data loss can occur, you are informed of the error and should take the required action.
Here is some code showing what happens when the receiver doesn't remove the data from it's queue. It also shows how much data can be queued up on the system running the receiver.
Sender:
#include <iostream>
#include <string>
#include <SFML/Network.hpp>
int main() {
sf::TcpSocket socket;
socket.connect( "12.34.56.78", 4422 );
socket.setBlocking( false );
std::size_t count = 0;
sf::Clock clock;
while( true ) {
sf::Packet packet;
std::string s( "0", 2048 );
packet << s;
sf::Socket::Status status = socket.send( packet );
if( status == sf::Socket::Done ) {
count += 2048;
std::cout << "Sent " << count << " bytes.\n";
} else if( status == sf::Socket::Error ) {
std::cout << "Error\n";
} else if( status == sf::Socket::Disconnected ) {
std::cout << "Disconnected\n";
} else if( status == sf::Socket::NotReady ) {
std::cout << "NotReady\n";
}
if( clock.getElapsedTime() > sf::seconds( 5 ) ) {
return 0;
}
}
return 0;
}
Receiver:
#include <iostream>
#include <SFML/Network.hpp>
int main() {
sf::TcpListener listener;
listener.listen( 4422 );
sf::TcpSocket socket;
listener.accept( socket );
sf::sleep( sf::seconds( 10 ) );
sf::Packet packet;
std::size_t count = 0;
while( !socket.receive( packet ) ) {
count += packet.getDataSize();
}
std::cout << "Dequeued " << count << " bytes.\n";
return 0;
}
Edit: If you need to take a break from coding you should watch a few good lecture videos about TCP. Here's one: http://www.youtube.com/watch?v=mHECg2Dt0Ao
-
After a lot of reviewing and labor, I can't reproduce the same errors again unless I use the exact networking structure I use in my project. Would it be okay if I posted it all here?
I will try to keep it as minimal as possible.
-
Ah ok, so after watching that video, the send() in the server will return NotReady if the receive buffer is full?
What if I already handle that? Here is my code for sending packets:
void Connector::sendPacket(sf::Packet* spack)
{
if(packets.size() <= 0)
{
sf::Socket::Status status = socket.send(*spack);
if(status == sf::Socket::NotReady || status == sf::Socket::Error)
{
packets.push(*spack);
if(status == sf::Socket::Error)
{
std::cout << "E1E\n";
}
}
}
else
{
packets.push(*spack);
}
}
and if it pushes the packet back, it'll eventually return with the retryDroppedPackets() function and do this:
if(!ent::connectors[i]->packets.empty())
{
sf::Socket::Status status = ent::connectors[i]->socket.send(ent::connectors[i]->packets.front());
if(status == sf::Socket::Done)
{
ent::connectors[i]->packets.pop();
}
if(status == sf::Socket::Error)
{
std::cout << "E1E\n";
}
}
Also, when the send() buffer is full like you said, would it only stop sending packets to ONE socket (the socket it was sending to)? Not the sockets of every person connected to the server?
Edit
I just read this over on GameDev.net
If [send()] doesn't block, it will buffer whatever it can and return that length.
So could this be the problem? Could the send() just be stuck with the overflow of data? Is there any way to avoid this?
-
Each connection has it's own buffers for sending and receiving. If one connection is stalled the others will not be affected.
I just read this over on GameDev.net
If [send()] doesn't block, it will buffer whatever it can and return that length.
So could this be the problem? Could the send() just be stuck with the overflow of data? Is there any way to avoid this?
There are only 2 things that can happen with your data when you call send(). Either it buffers it for sending, or it doesn't. It doesn't matter whether the remote host has space or not, it will still be buffered in the send buffer because this is required by TCP. When the data isn't buffered, the send() call either blocks if you are in blocking mode, or it returns NotReady in non-blocking mode. The send() call doesn't get "stuck".
Actually... I just noticed something in the SFML source code... going to investigate.
Edit: Yeah it seems there is a really nasty bug that is causing this ::). I'll start a separate thread when I get home today.
-
I think delaying the sending of large packets quickly prevents the problem from happening.
EDIT:
Nope.
-
Edit: Yeah it seems there is a really nasty bug that is causing this ::). I'll start a separate thread when I get home today.
Ah, finally! :D
-
http://en.sfml-dev.org/forums/index.php?topic=11780.0
Let's leave this thread to the necromancers who will lurk these forums sometime in the future :).