SFML community forums

Help => Network => Topic started by: Deliciouscookie on June 22, 2018, 08:52:05 am

Title: VoIp System where server passes clients speech to every other client.
Post by: Deliciouscookie on June 22, 2018, 08:52:05 am

Hello everybody. I am working with a Virtual Construction and we are trying to create a VoIP system to our Unreal Engine Project. We added the Audio libraries and managed to record and play the Microphone sound. However, we are unsuccessful to make Voice chat working as described in Topic. What we are are trying to achieve is that when one client speaks, it will pass the voice data to Dedicated Server than then multicast it to every other client. Basically basic voice chat use see in almost every multiplayer game.

There is VoIP example and it uses own Network Sound class, but it seems, that is only connects to one IP and when connected, no other clients can connect it, so it might not be suitable in this scenario. Any help from you guys? Thank you very much.

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: eXpl0it3r on June 22, 2018, 10:42:39 am

So with what are you exactly stuck?

In order to send data to multiple clients you of course have to open a connection to each client.
Have you implemented this already?

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Deliciouscookie on June 22, 2018, 02:58:15 pm

Quote from: eXpl0it3r on June 22, 2018, 10:42:39 am

So with what are you exactly stuck?

In order to send data to multiple clients you of course have to open a connection to each client.
Have you implemented this already?

For what I undestand, when you open connection, it can only take one connection and then closes to ther connections. So what if there is a ten people playing? So do I need to create ten listeners and connect every client to each listener? Project is also drop in and drop out system, so would I require open listening connection for player when joining game and then make all clients to connect the joined person? Thanks.

edit: Also, it would not be good if clients need to open the listening port, because we might not be sure, if the player has the port blocked or already used. So in this case the server needs to listen evereybody and then forward the data to clients. So clients would need some other system that plays the sound as the server passes the data. So I need help with that.

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Laurent on June 22, 2018, 03:30:45 pm

Quote

For what I undestand, when you open connection, it can only take one connection and then closes to ther connections.

That's wrong, of course. A listener can accept more than one connection, and once accepted, a connection can live as long as you need it. Nothing forces you to have a single connection active at a time.

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Deliciouscookie on June 22, 2018, 03:49:16 pm

Quote from: Laurent on June 22, 2018, 03:30:45 pm

Quote

For what I undestand, when you open connection, it can only take one connection and then closes to ther connections.

That's wrong, of course. A listener can accept more than one connection, and once accepted, a connection can live as long as you need it. Nothing forces you to have a single connection active at a time.

Hmm. Interesting. So that would solve one problem. So in Dedicated server, I would open listener. And then all clients connect to that one listener. And when server gets the data, instead of playing it, it would just send it to all clients.

So would I pass the data.samples from this function to clients (code taken from Server.cpp in examples)?

Quote

    virtual bool onGetData(sf::SoundStream::Chunk& data)
    {
        // We have reached the end of the buffer and all audio data have been played: we can stop playback
        if ((m_offset >= m_samples.size()) && m_hasFinished)
            return false;

        // No new data has arrived since last update: wait until we get some
        while ((m_offset >= m_samples.size()) && !m_hasFinished)
            sf::sleep(sf::milliseconds(10));

        // Copy samples into a local buffer to avoid synchronization problems
        // (don't forget that we run in two separate threads)
        {
            sf::Lock lock(m_mutex);
            m_tempBuffer.assign(m_samples.begin() + m_offset, m_samples.end());
        }

        // Fill audio data to pass to the stream
        data.samples     = &m_tempBuffer[0];
        data.sampleCount = m_tempBuffer.size();

        // Update the playing offset
        m_offset += m_tempBuffer.size();

        return true;
    }

I assume there is a simple function that can be run all the time from clients that takes the data when passed to them? This is getting interesting.

edit: Unless... Does every client hear the others when connected to same listener automatically? That would solve lot of things and save me from passing things from the server side manually. Sorry for questions,  I would love to test this myself, but for now, I only have single computer and cannot try anything networked until far next week :).

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Laurent on June 22, 2018, 04:28:11 pm

Quote

Does every client hear the others when connected to same listener automatically?

No, a connection is between two peers. Otherwise you have to use UDP broadcast.

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Deliciouscookie on June 24, 2018, 11:36:57 am

Quote from: Laurent on June 22, 2018, 03:30:45 pm

Quote

For what I undestand, when you open connection, it can only take one connection and then closes to ther connections.

That's wrong, of course. A listener can accept more than one connection, and once accepted, a connection can live as long as you need it. Nothing forces you to have a single connection active at a time.

Thanks for reply. However, I cannnot get two clients to connect to a listening server. When ever I connect one client, everything works. When I connect second client, the server won't take the connection. I tried to look up the code and make different kind of modifications, but result is same. Two client are unable to connect to server. If I open two connections with two different ports then everything of course works, but that isn't ideal at all :/..

Any help guys?
Here is the example code I used: https://github.com/SFML/SFML/tree/master/examples/voip

edit: I have tried this only in local network. Atleast then, Server only says "Client Connected" for the first clients and no others. Not sure If I create a connection over internet.

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Laurent on June 24, 2018, 08:55:49 pm

This example doesn't manage multiple clients, it stops listening after the first connection, on purpose. So you can't just use it as it is and hope that it will work for your use case, you'll have to design a multi-client server by yourself (it's not that hard... just have listener.accept in a loop).

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Deliciouscookie on June 25, 2018, 12:10:47 pm

Quote from: Laurent on June 24, 2018, 08:55:49 pm

This example doesn't manage multiple clients, it stops listening after the first connection, on purpose. So you can't just use it as it is and hope that it will work for your use case, you'll have to design a multi-client server by yourself (it's not that hard... just have listener.accept in a loop).

Thanks for the comment Laurent. After lot of studying, I managed to get this further. Now my server will listen and accept multiple client. My Server code is:

(click to show/hide)

Quote

////////////////////////////////////////////////////////////
const sf::Uint8 audioData = 1;
const sf::Uint8 endOfStream = 2;

/// Customized sound stream for acquiring audio data
/// from the network
////////////////////////////////////////////////////////////

class NetworkAudioStream : public sf::SoundStream
{

public:
   ////////////////////////////////////////////////////////////
   /// Default constructor
   ///
   ////////////////////////////////////////////////////////////
   NetworkAudioStream() :
      m_offset(0),
      m_hasFinished(false)
   {
      // Set the sound parameters
      initialize(1, 44100);
   }

   ////////////////////////////////////////////////////////////
   /// Run the server, stream audio data from the client
   ///
   ////////////////////////////////////////////////////////////
   void start(unsigned short port)
   {
         m_listener.listen(port);
         Selector.add(m_listener);

         // Start playback
         play();

         // Start receiving audio data
         receiveLoop();
   }

private:

   ////////////////////////////////////////////////////////////
   /// /see SoundStream::OnGetData
   ///
   ////////////////////////////////////////////////////////////
   virtual bool onGetData(sf::SoundStream::Chunk& data)
   {
      // We have reached the end of the buffer and all audio data have been played: we can stop playback
      if ((m_offset >= m_samples.size()) && m_hasFinished)
         return false;

      // No new data has arrived since last update: wait until we get some
      while ((m_offset >= m_samples.size()) && !m_hasFinished)
         sf::sleep(sf::milliseconds(10));

      // Copy samples into a local buffer to avoid synchronization problems
      // (don't forget that we run in two separate threads)
      {
         sf::Lock lock(m_mutex);
         m_tempBuffer.assign(m_samples.begin() + m_offset, m_samples.end());
      }

      // Fill audio data to pass to the stream
      data.samples = &m_tempBuffer[0];
      data.sampleCount = m_tempBuffer.size();

      Buffer.loadFromMemory(&m_tempBuffer[0], m_tempBuffer.size());
      TestDelegate.ExecuteIfBound(Buffer);
      SendSoundDelegate.Broadcast(true);

      // Update the playing offset
      m_offset += m_tempBuffer.size();

      return true;
   }

   ////////////////////////////////////////////////////////////
   /// /see SoundStream::OnSeek
   ///
   ////////////////////////////////////////////////////////////
   virtual void onSeek(sf::Time timeOffset)
   {
      m_offset = timeOffset.asMilliseconds() * getSampleRate() * getChannelCount() / 1000;
   }

   ////////////////////////////////////////////////////////////
   /// Get audio data from the client until playback is stopped
   ///
   ////////////////////////////////////////////////////////////
   void receiveLoop()
   {   
      // Endless loop that waits for new connections
      while (true)
      {
         // Make the selector wait for data on any socket
         if (Selector.wait())
         {
            // Test the listener
            if (Selector.isReady(m_listener))
            {
               UE_LOG(LogTemp, Warning, TEXT("Selector is ready"));
               // The listener is ready: there is a pending connection
               sf::TcpSocket* client = new sf::TcpSocket;
               if (m_listener.accept(*client) == sf::Socket::Done)
               {                  
                  // Add the new client to the clients list
                  clients.push_back(client);
                  // Add the new client to the selector so that we will
                  // be notified when he sends something
                  Selector.add(*client);
                  UE_LOG(LogTemp, Warning, TEXT("Client Joined"));
               }
               else
               {
                  // Error, we won't get a new connection, delete the socket
                  delete client;
               }
            }
            else
            {
               // The listener socket is not ready, test all other sockets (the clients)
               for (std::list<sf::TcpSocket*>::iterator it = clients.begin(); it != clients.end(); ++it)
               {
                  sf::TcpSocket& client = **it;
                  if (Selector.isReady(client))
                  {
                     // The client has sent some data, we can receive it
                     sf::Packet packet;
                     if (client.receive(packet) == sf::Socket::Done)
                     {
                        // Get waiting audio data from the network
                        sf::Packet packet;
                        if (client.receive(packet) != sf::Socket::Done)
                           break;

                        // Extract the message ID
                        sf::Uint8 id;
                        packet >> id;

                        if (id == audioData)
                        {
                           // Extract audio samples from the packet, and append it to our samples buffer
                           const sf::Int16* samples = reinterpret_cast<const sf::Int16*>(static_cast<const char*>(packet.getData()) + 1);
                           std::size_t      sampleCount = (packet.getDataSize() - 1) / sizeof(sf::Int16);

                           // Don't forget that the other thread can access the sample array at any time
                           // (so we protect any operation on it with the mutex)
                           {
                              sf::Lock lock(m_mutex);
                              std::copy(samples, samples + sampleCount, std::back_inserter(m_samples));
                           }
                        }
                        else if (id == endOfStream)
                        {
                           // End of stream reached: we stop receiving audio data
                           //std::cout << "Audio data has been 100% received!" << std::endl;
                        }
                        else
                        {
                           // Something's wrong...
                           //std::cout << "Invalid packet received..." << std::endl;


                        }
                     }
                  }
               }
            }
         }
      }
   }

   ////////////////////////////////////////////////////////////
   // Member data
   ////////////////////////////////////////////////////////////
   sf::TcpListener        m_listener;
   // Create a list to store the future clients
   std::list<sf::TcpSocket*> clients;
   sf::SocketSelector     Selector;


   sf::Mutex              m_mutex;
   std::vector<sf::Int16> m_samples;
   std::vector<sf::Int16> m_tempBuffer;
   std::size_t            m_offset;
   bool                   m_hasFinished;


public:
   FHasClientJoined      ClientJoinedDelegate;
   sf::SoundBuffer         Buffer;
   FSendSound            SendSoundDelegate;
   FMyDelegate            TestDelegate;
};

Client is the same as before:

(click to show/hide)

Quote

////////////////////////////////////////////////////////////
// Headers
////////////////////////////////////////////////////////////
#include <SFML/Audio.hpp>
#include <SFML/Network.hpp>
#include <iostream>


const sf::Uint8 audioData   = 1;
const sf::Uint8 endOfStream = 2;


////////////////////////////////////////////////////////////
/// Specialization of audio recorder for sending recorded audio
/// data through the network
////////////////////////////////////////////////////////////
class NetworkRecorder : public sf::SoundRecorder
{
public:

    ////////////////////////////////////////////////////////////
    /// Constructor
    ///
    /// \param host Remote host to which send the recording data
    /// \param port Port of the remote host
    ///
    ////////////////////////////////////////////////////////////
    NetworkRecorder(const sf::IpAddress& host, unsigned short port) :
    m_host(host),
    m_port(port)
    {
    }

    ////////////////////////////////////////////////////////////
    /// Destructor
    ///
    /// \see SoundRecorder::~SoundRecorder()
    ///
    ////////////////////////////////////////////////////////////
    ~NetworkRecorder()
    {
        // Make sure to stop the recording thread
        stop();
    }

private:

    ////////////////////////////////////////////////////////////
    /// \see SoundRecorder::onStart
    ///
    ////////////////////////////////////////////////////////////
    virtual bool onStart()
    {
        if (m_socket.connect(m_host, m_port) == sf::Socket::Done)
        {
            std::cout << "Connected to server " << m_host << std::endl;
            return true;
        }
        else
        {
            return false;
        }
    }

    ////////////////////////////////////////////////////////////
    /// \see SoundRecorder::onProcessSamples
    ///
    ////////////////////////////////////////////////////////////
    virtual bool onProcessSamples(const sf::Int16* samples, std::size_t sampleCount)
    {
        // Pack the audio samples into a network packet
        sf::Packet packet;
        packet << audioData;
        packet.append(samples, sampleCount * sizeof(sf::Int16));

        // Send the audio packet to the server
        return m_socket.send(packet) == sf::Socket::Done;
    }

    ////////////////////////////////////////////////////////////
    /// \see SoundRecorder::onStop
    ///
    ////////////////////////////////////////////////////////////
    virtual void onStop()
    {
        // Send a "end-of-stream" packet
        sf::Packet packet;
        packet << endOfStream;
        m_socket.send(packet);

        // Close the socket
        m_socket.disconnect();
    }

    ////////////////////////////////////////////////////////////
    // Member data
    ////////////////////////////////////////////////////////////
    sf::IpAddress  m_host;   ///< Address of the remote host
    unsigned short m_port;   ///< Remote port
    sf::TcpSocket  m_socket; ///< Socket used to communicate with the server
};


////////////////////////////////////////////////////////////
/// Create a client, connect it to a running server and
/// start sending him audio data
///
////////////////////////////////////////////////////////////
void doClient(unsigned short port)
{
    // Check that the device can capture audio
    if (!sf::SoundRecorder::isAvailable())
    {
        std::cout << "Sorry, audio capture is not supported by your system" << std::endl;
        return;
    }

    // Ask for server address
    sf::IpAddress server;
    do
    {
        std::cout << "Type address or name of the server to connect to: ";
        std::cin  >> server;
    }
    while (server == sf::IpAddress::None);

    // Create an instance of our custom recorder
    NetworkRecorder recorder(server, port);

    // Wait for user input...
    std::cin.ignore(10000, '\n');
    std::cout << "Press enter to start recording audio";
    std::cin.ignore(10000, '\n');

    // Start capturing audio data
    recorder.start(44100);
    std::cout << "Recording... press enter to stop";
    std::cin.ignore(10000, '\n');
    recorder.stop();
}

My Problem is that sound sounds really weird! It's kinda cuts of sometimes and almost all the time, I can hear my self twice. I even set up a local network where other computer was a server and then I connected to it through another computer. So it was not just a circular sound problem that was happening.

So I am having a hard time with the twice playing sound AND I am not really sure how to go on, if I want to send the sound to all clients that are connected to server  :-\? Any help with these two again, thank you.

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Laurent on June 25, 2018, 12:33:10 pm

Quote

My Problem is that sound sounds really weird!

Was it already like this with the unmodified code (with a single client)? Or did it stop working fine after your modifications?

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Deliciouscookie on June 25, 2018, 02:44:26 pm

Quote from: Laurent on June 25, 2018, 12:33:10 pm

Quote

My Problem is that sound sounds really weird!

Was it already like this with the unmodified code (with a single client)? Or did it stop working fine after your modifications?

It was same before. Modified code just allowed additional clients to join, but didn't improve the quality itself.

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Laurent on June 25, 2018, 03:09:43 pm

The example should work fine, but if it doesn't, then you should clearly not base your code on it. It will probably be harder and take more time, but maybe you should start from scratch; you would learn more about all this network stuff and in the end, understand better what's going on in your program.

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Deliciouscookie on June 25, 2018, 03:30:41 pm

Quote from: Laurent on June 25, 2018, 03:09:43 pm

The example should work fine, but if it doesn't, then you should clearly not base your code on it. It will probably be harder and take more time, but maybe you should start from scratch; you would learn more about all this network stuff and in the end, understand better what's going on in your program.

I guess you are right. Maybe there is something wrong with implementing the code into Unreal Engine. I will look upon that. Perhaps Unreal own multi-threading is messing something up. I need to look upon these. Thanks for your help. But hey! About the last question that was on my mind. Any suggestions how to start with server sending the voip to all client when one speaks?

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Laurent on June 25, 2018, 04:02:55 pm

Quote

About the last question that was on my mind. Any suggestions how to start with server sending the voip to all client when one speaks?

Once you have a server that accepts multiple clients, what's the problem with sending the data to all of them? Where are you stuck?

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Deliciouscookie on June 25, 2018, 06:26:05 pm

Quote from: Laurent on June 25, 2018, 04:02:55 pm

Quote

About the last question that was on my mind. Any suggestions how to start with server sending the voip to all client when one speaks?

Once you have a server that accepts multiple clients, what's the problem with sending the data to all of them? Where are you stuck?

Good news! I found a bug in my code and now voice is working perfectly with multiple clients! I'm sorry if I sound stupid. My Speciality is with Unreal Engine and I am still somewhat unfamiliar with networking this SFML provides. So where I am stuck is, I'm not really sure what I am supposed to do all together. I could maybe use Unreal Engines networking properties to multicast the Received Packet to clients and then put it in buffer and try to play it, but I am pretty sure, that it would be much better to do here in SFML and leave Unreal out of it as much as possible. I really trying to learn at the same time, so bare with me here  ::)

Here is the working code that is little cleaned up and the faulty break;s have been removed.

(click to show/hide)

Quote

const sf::Uint8 audioData = 1;
const sf::Uint8 endOfStream = 2;


////////////////////////////////////////////////////////////
/// Customized sound stream for acquiring audio data
/// from the network
////////////////////////////////////////////////////////////
class NetworkAudioStream : public sf::SoundStream
{
public:

   ////////////////////////////////////////////////////////////
   /// Default constructor
   ///
   ////////////////////////////////////////////////////////////
   NetworkAudioStream() :
      m_offset(0),
      m_hasFinished(false)
   {
      // Set the sound parameters
      initialize(1, 44100);
   }

   ////////////////////////////////////////////////////////////
   /// Run the server, stream audio data from the client
   ///
   ////////////////////////////////////////////////////////////
   void start(unsigned short port)
   {
         // Listen to the given port for incoming connections
         if (m_listener.listen(port) != sf::Socket::Done)
            return;
         UE_LOG(LogTemp, Warning, TEXT("Server Started at port %i"), port);

         selector.add(m_listener);

         // Start playback
         play();

         // Start receiving audio data
         receiveLoop();
   }

private:

   ////////////////////////////////////////////////////////////
   /// /see SoundStream::OnGetData
   ///
   ////////////////////////////////////////////////////////////
   virtual bool onGetData(sf::SoundStream::Chunk& data)
   {
      // We have reached the end of the buffer and all audio data have been played: we can stop playback
      if ((m_offset >= m_samples.size()) && m_hasFinished)
         return false;

      // No new data has arrived since last update: wait until we get some
      while ((m_offset >= m_samples.size()) && !m_hasFinished)
         sf::sleep(sf::milliseconds(10));

      // Copy samples into a local buffer to avoid synchronization problems
      // (don't forget that we run in two separate threads)
      {
         sf::Lock lock(m_mutex);
         m_tempBuffer.assign(m_samples.begin() + m_offset, m_samples.end());
      }

      // Fill audio data to pass to the stream
      data.samples = &m_tempBuffer[0];
      data.sampleCount = m_tempBuffer.size();

      // Update the playing offset
      m_offset += m_tempBuffer.size();

      return true;
   }

   ////////////////////////////////////////////////////////////
   /// /see SoundStream::OnSeek
   ///
   ////////////////////////////////////////////////////////////
   virtual void onSeek(sf::Time timeOffset)
   {
      m_offset = timeOffset.asMilliseconds() * getSampleRate() * getChannelCount() / 1000;
   }

   ////////////////////////////////////////////////////////////
   /// Get audio data from the client until playback is stopped
   ///
   ////////////////////////////////////////////////////////////
   void receiveLoop()
   {
      while (true)
      {
         // Make the selector wait for data on any socket
         if (selector.wait())
         {
            // Test the listener
            if (selector.isReady(m_listener))
            {
               // The listener is ready: there is a pending connection
               sf::TcpSocket* client = new sf::TcpSocket;
               if (m_listener.accept(*client) == sf::Socket::Done)
               {
                  // Add the new client to the clients list
                  clients.push_back(client);
                  // Add the new client to the selector so that we will
                  // be notified when he sends something
                  selector.add(*client);
                  UE_LOG(LogTemp, Warning, TEXT("New Client Joined to chat"));
               }
               else
               {
                  // Error, we won't get a new connection, delete the socket
                  delete client;
               }
            }
            else
            {
               // The listener socket is not ready, test all other sockets (the clients)
               for (std::list<sf::TcpSocket*>::iterator it = clients.begin(); it != clients.end(); ++it)
               {
                  sf::TcpSocket& client = **it;
                  if (selector.isReady(client))
                  {
                     // The client has sent some data, we can receive it
                     sf::Packet packet;
                     if (client.receive(packet) == sf::Socket::Done)
                     {
                        // Get waiting audio data from the network
                        //sf::Packet packet;
                        //if (client.receive(packet) != sf::Socket::Done)
                        //   break;

                        // Extract the message ID
                        sf::Uint8 id;
                        packet >> id;

                        if (id == audioData)
                        {
                           // Extract audio samples from the packet, and append it to our samples buffer
                           const sf::Int16* samples = reinterpret_cast<const sf::Int16*>(static_cast<const char*>(packet.getData()) + 1);
                           std::size_t      sampleCount = (packet.getDataSize() - 1) / sizeof(sf::Int16);

                           // Don't forget that the other thread can access the sample array at any time
                           // (so we protect any operation on it with the mutex)
                           {
                              sf::Lock lock(m_mutex);
                              std::copy(samples, samples + sampleCount, std::back_inserter(m_samples));
                           }
                        }
                        else if (id == endOfStream)
                        {
                           // End of stream reached: we stop receiving audio data
                           //   std::cout << "Audio data has been 100% received!" << std::endl;
                           UE_LOG(LogTemp, Warning, TEXT("Audio Stream successfully finished"));
                        }
                        else
                        {
                           // Something's wrong...
                           UE_LOG(LogTemp, Warning, TEXT("Invalid packet received..."));

                        }
                     }
                  }
               }
            }
         }
      }
   }

   ////////////////////////////////////////////////////////////
   // Member data
   ////////////////////////////////////////////////////////////
   sf::TcpListener        m_listener;
   sf::TcpSocket          m_client;
   // Create a list to store the future clients
   std::list<sf::TcpSocket*> clients;
   // Create a selector
   sf::SocketSelector selector;

   sf::Mutex              m_mutex;
   std::vector<sf::Int16> m_samples;
   std::vector<sf::Int16> m_tempBuffer;
   std::size_t            m_offset;
   bool                   m_hasFinished;
};

edit: I think sending the packet here to other clients is the key. I was wondering to use something like this after Server is sure it received the voice package:

Quote

for (std::list<sf::TcpSocket*>::iterator it = clients.begin(); it != clients.end(); ++it)
{
   sf::TcpSocket& client2 = **it;
        // Don't send same voice to speaker.
   if (client2.getLocalPort() != client.getLocalPort())
   {
      UE_LOG(LogTemp, Warning, TEXT("Sending audio to one of the clients"));
      client2.send(packet);
   }
}

But if I use that, i'm not still sure how the clients can receive the packet. Am I in the right tracks here? Any help about receiving the packet on the client side? Thanks.

Above code would go inside this IF-sentence

Quote

if (id == audioData)
       {
 // Extract audio samples from the packet, and append it to our samples buffer
          const sf::Int16* samples = reinterpret_cast<const sf::Int16*>(static_cast<const char*>(packet.getData()) + 1);

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Laurent on June 26, 2018, 08:31:22 am

Yes, that looks correct.

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Deliciouscookie on June 26, 2018, 08:16:43 pm

Quote from: Laurent on June 26, 2018, 08:31:22 am

Yes, that looks correct.

So if client is connected to Server. Do I need to make some forever running loop in clients also? And If client is connected, does it receive packages automatically if I use something like this:

Quote

char data[100];
std::size_t received;

// TCP socket:
if (socket.receive(data, 100, received) != sf::Socket::Done)
{
    // error...
}
std::cout << "Received " << received << " bytes" << std::endl;

Or what kind of procedures I need to set in client side, so it would get the packed data from server? I am lost because server created the NetworkAudioStream from SoundStream, so do I need to create same stream to client also or how does it work in Server--> Client side? .

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Laurent on June 27, 2018, 07:55:55 am

You should go step by step. Do simple networking first, don't try to do everything at the same time. We could help you write your program on this forum, but you wouldn't learn much in the end. Read the tutorials, write simple client / server programs, test things by yourself, etc. :)

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Deliciouscookie on June 28, 2018, 10:13:54 pm

--- Nothing Yet. Something happening.

Edit: Yesss. It's working finally! Phew, I thought I never see end of it. Now it's time to take deep breath, import this whole system as Unreal Engine Plugin, test this for good and then starting to figure out how can I detect when people is speaking and just then send the audio over network. Also need to figure out how I can pass the data of the person that is speaking to Unreal so the avatar can me marked as speaking avatar.

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Deliciouscookie on July 03, 2018, 05:18:03 pm

One wall came up again and not been able to solve it.

I have two classes.

(click to show/hide)

Quote

.H

// Fill out your copyright notice in the Description page of Project Settings.

#pragma once

#include "CoreMinimal.h"
#include "UObject/NoExportTypes.h"
#include "SFML/Audio.hpp"
#include <SFML/Network.hpp>
#include <list>
#include "HAL/ThreadSafeBool.h"
#include "Public/TimerManager.h"
#include "Engine/World.h"
#include "sfmlServer.generated.h"






////////////////////////////////////////////////////////////
/// Customized sound stream for acquiring audio data
/// from the network
////////////////////////////////////////////////////////////
class NetworkAudioStream : public sf::SoundStream
{
public:
   const sf::Uint8 audioData = 1;
   const sf::Uint8 endOfStream = 2;
   ////////////////////////////////////////////////////////////
   /// Default constructor
   ///
   ////////////////////////////////////////////////////////////
   NetworkAudioStream() :
      m_offset(0),
      m_hasFinished(false)
   {
      // Set the sound parameters
      initialize(1, 44100);
   }

   ////////////////////////////////////////////////////////////
   /// Run the server, stream audio data from the client
   ///
   ////////////////////////////////////////////////////////////
   void start(unsigned short port)
   {
      // Listen to the given port for incoming connections
      if (m_listener.listen(port) != sf::Socket::Done)
         return;
      UE_LOG(LogTemp, Warning, TEXT("Server Started at port %i"), port);

      selector.add(m_listener);

      // Start playback
      play();

      // Start receiving audio data
      receiveLoop();
   }

private:

   ////////////////////////////////////////////////////////////
   /// /see SoundStream::OnGetData
   ///
   ////////////////////////////////////////////////////////////
   virtual bool onGetData(sf::SoundStream::Chunk& data)
   {
      // We have reached the end of the buffer and all audio data have been played: we can stop playback
      if ((m_offset >= m_samples.size()) && m_hasFinished)
         return false;

      // No new data has arrived since last update: wait until we get some
      while ((m_offset >= m_samples.size()) && !m_hasFinished)
         sf::sleep(sf::milliseconds(10));

      // Copy samples into a local buffer to avoid synchronization problems
      // (don't forget that we run in two separate threads)
      {
         sf::Lock lock(m_mutex);
         m_tempBuffer.assign(m_samples.begin() + m_offset, m_samples.end());
      }

      // Fill audio data to pass to the stream
      data.samples = &m_tempBuffer[0];
      data.sampleCount = m_tempBuffer.size();

      // Update the playing offset
      m_offset += m_tempBuffer.size();
      UE_LOG(LogTemp, Warning, TEXT("Getting Data"));
      return true;
   }

   ////////////////////////////////////////////////////////////
   /// /see SoundStream::OnSeek
   ///
   ////////////////////////////////////////////////////////////
   virtual void onSeek(sf::Time timeOffset)
   {
      m_offset = timeOffset.asMilliseconds() * getSampleRate() * getChannelCount() / 1000;
   }

   ////////////////////////////////////////////////////////////
   /// Get audio data from the client until playback is stopped
   ///
   ////////////////////////////////////////////////////////////
   void receiveLoop()
   {         
      while (Canrun)
      {
         // Make the selector wait for data on any socket
         if (selector.wait())
         {
            // Test the listener
            if (selector.isReady(m_listener))
            {
               // The listener is ready: there is a pending connection
               sf::TcpSocket* client = new sf::TcpSocket;
               if (m_listener.accept(*client) == sf::Socket::Done)
               {   
                  // Add the new client to the clients list
                  clients.push_back(client);
                  // Add the new client to the selector so that we will
                  // be notified when he sends something
                  selector.add(*client);
                  UE_LOG(LogTemp, Warning, TEXT("New Client Joined to chat"));
               }
               else
               {
                  // Error, we won't get a new connection, delete the socket
                  delete client;
               }
            }
            else
            {
               // The listener socket is not ready, test all other sockets (the clients)
               for (std::list<sf::TcpSocket*>::iterator it = clients.begin(); it != clients.end(); ++it)
               {
                  sf::TcpSocket& client = **it;
                  if (selector.isReady(client))
                  {
                     // The client has sent some data, we can receive it
                     sf::Packet packet;
                     if (client.receive(packet) == sf::Socket::Done)
                     {
                        // Get waiting audio data from the network
                        //sf::Packet packet;
                        //if (client.receive(packet) != sf::Socket::Done)
                        //   break;

                        // Extract the message ID
                        sf::Uint8 id;
                        packet >> id;


                        if (id == endOfStream)
                        {
                           // End of stream reached: we stop receiving audio data
                           //   std::cout << "Audio data has been 100% received!" << std::endl;
                           UE_LOG(LogTemp, Warning, TEXT("Audio Stream successfully finished"));
                           stop();
                           Canrun = false;
                        }

                     else if (id == audioData)
                        {
                           // Extract audio samples from the packet, and append it to our samples buffer
                           const sf::Int16* samples = reinterpret_cast<const sf::Int16*>(static_cast<const char*>(packet.getData()) + 1);
                           std::size_t      sampleCount = (packet.getDataSize() - 1) / sizeof(sf::Int16);

                           // Don't forget that the other thread can access the sample array at any time
                           // (so we protect any operation on it with the mutex)
                           {
                              sf::Lock lock(m_mutex);
                              std::copy(samples, samples + sampleCount, std::back_inserter(m_samples));
                           }


                           // Send to all clients.
                           
                           for (std::list<sf::TcpSocket*>::iterator it = clients.begin(); it != clients.end(); ++it)
                           {
                              sf::TcpSocket& client2 = **it;
                              // Don't send same voice to speaker.
                              if (client2.getRemoteAddress() != client.getRemoteAddress())
                              {
                                 client2.send(packet);
                              }
                           }
                        }
                     
   
                        else
                        {
                           // Something's wrong...
                           UE_LOG(LogTemp, Warning, TEXT("Invalid packet received..."));
                           stop();
                           Canrun = false;

                        }
                     }
                  }
               }
            }
         }
      }
      UE_LOG(LogTemp, Warning, TEXT("Exit While Loop"));
   }

   ////////////////////////////////////////////////////////////
   // Member data
   ////////////////////////////////////////////////////////////
   sf::TcpListener        m_listener;
   sf::TcpSocket          m_client;
   // Create a list to store the future clients
   std::list<sf::TcpSocket*> clients;
   // Create a selector
   sf::SocketSelector selector;

   sf::Mutex              m_mutex;
   std::vector<sf::Int16> m_samples;
   std::vector<sf::Int16> m_tempBuffer;
   std::size_t            m_offset;
   bool                   m_hasFinished;
public:
   bool Canrun = true;
};




UCLASS()
class SFMLTEST_API UsfmlServer : public UObject
{
   GENERATED_BODY()

public:
   NetworkAudioStream audioStream;
   
   UFUNCTION(BlueprintCallable)
   void StartServer(int32 port);

   UFUNCTION(BlueprintCallable)
   void StopServer();   
};

As you can see this class is responsible for listening clients and passing sound packets to everybody. Now hear me out, because this sounds so strange to me and I think there is some underlying thing that I don't know. I have problem by having this class to go full circle and closing.

I can start the Server in my project and Clients can join to it. If client leaves and sends the endOfStream my server successfully completes and shuts down. All good so far.

Problem is that I want to shutdown server in my own computer. Logically I could simply change the Canrun to false and program would shutdown in next loop but that ain't happening. Program gets stuck to somewhere. I even created custom class and send the same endOfStream message to server locally, but that ain't helping, the class just wont complete. I then tried to use the example VoIP in same computer where the server is running by connecting to localhost. After sending endOfStream, the server shuts down propertly again.

Now comes the strange part. While I was debugging, I removed EVERYTHING from the receiveLoop() function and still the class wont complete. The only way I get it to complete is to connect to it from different program and send the end message. I have been fighting this for so long but cannot figure out what is causing this.

My .CPP file here:
This is my current attempt. But like I said, I had a system, that spawn a temporaly class that sends the EndOfStream Packet to server from the same computer, but that didn't help. I can see that server receives the packet. and says UE_LOG(LogTemp, Warning, TEXT("Audio Stream successfully finished")); but the program wont end. Also, notice that I need to run this program in backgroundthread in Unrea or the mainthread freezes. Not sure if that has something to do with it.

Quote

(click to show/hide)

// Fill out your copyright notice in the Description page of Project Settings.

#include "sfmlServer.h"
#include "LambdaRunnable.h"

FLambdaRunnable* LambdaServer;


////////////////////////////////////////////////////////////
/// Launch a server and wait for incoming audio data from
/// a connected client
///
////////////////////////////////////////////////////////////
void UsfmlServer::StartServer(int32 port)
{
   LambdaServer = FLambdaRunnable::RunLambdaOnBackGroundThread([&]()
   {
      // Build an audio stream to play sound data as it is received through the network
      audioStream.start(port);

      // Loop until the sound playback is finished
      while (audioStream.getStatus() != sf::SoundStream::Stopped)
      {
         // Leave some CPU time for other threads
         sf::sleep(sf::milliseconds(100));
         //Initial wait before starting
         FPlatformProcess::Sleep(100);
      }
   });
}

void UsfmlServer::StopServer()
{
   audioStream.Canrun = false;
}

I'm all out of hope here. What bugs me the most is WHY it's working as it supposed to when the data is send over the network, but not locally. And why the program doesn't complete, even if the whole function is empty. Please help.

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: eXpl0it3r on July 03, 2018, 05:41:53 pm

You're using blocking sockets, so the receive calls will block until they receive data. Which also means that your function gets stuck waiting for data.

Maybe there are more issues that I missed by glancing over it.

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Deliciouscookie on July 03, 2018, 07:49:41 pm

Quote from: eXpl0it3r on July 03, 2018, 05:41:53 pm

You're using blocking sockets, so the receive calls will block until they receive data. Which also means that your function gets stuck waiting for data.

Maybe there are more issues that I missed by glancing over it.

I think I found out the culprit. It is because stop() function won't work if the EndStream data won't come over network. I tested this by doing this:

Quote

                        if (id == endOfStream)
                        {
                           // End of stream reached: we stop receiving audio data
                           //   std::cout << "Audio data has been 100% received!" << std::endl;
                           stop();
                           Canrun = false;
                           UE_LOG(LogTemp, Warning, TEXT("Audio Stream successfully finished"));
                        }

If I create server and connect client through other program and then send endOfStream id, program goes pass the stop() and I get the LOG printing. If I create client and connect it through the visual studio code and then send endOfStream id, I never get the LOG printing, so program halts to stop and never goes pass it. Also, If I remove all the code and not run the stop(), the program never ends. The problem was never that I won't getting out of the loop. Problem is that I cannot run the stop() command for some reason.

So, question is, why the stop() won't run for me.

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Deliciouscookie on July 03, 2018, 08:42:55 pm

Quote from: Deliciouscookie on July 03, 2018, 07:49:41 pm

Quote from: eXpl0it3r on July 03, 2018, 05:41:53 pm

You're using blocking sockets, so the receive calls will block until they receive data. Which also means that your function gets stuck waiting for data.

Maybe there are more issues that I missed by glancing over it.

I think I found out the culprit. It is because stop() function won't work if the EndStream data won't come over network. I tested this by doing this:

Quote

                        if (id == endOfStream)
                        {
                           // End of stream reached: we stop receiving audio data
                           //   std::cout << "Audio data has been 100% received!" << std::endl;
                           stop();
                           Canrun = false;
                           UE_LOG(LogTemp, Warning, TEXT("Audio Stream successfully finished"));
                        }

If I create server and connect client through other program and then send endOfStream id, program goes pass the stop() and I get the LOG printing. If I create client and connect it through the visual studio code and then send endOfStream id, I never get the LOG printing, so program halts to stop and never goes pass it. Also, If I remove all the code and not run the stop(), the program never ends. The problem was never that I won't getting out of the loop. Problem is that I cannot run the stop() command for some reason.

So, question is, why the stop() won't run for me.

Got it working. Problem was when I spawned the class that sole purpose was to send the endStream message, it was destroyed too soon. So I added a little timer so the buffer get some data and then the endStream message goes through and everything closes nicely. Not best way i'm sure, but atleast now it's working without constant crashes

Quote

   FTimerDelegate TimerCallback;
   TimerCallback.BindLambda([&]
   {
      WaitLittle = false;
   });

   FTimerHandle Handle;
   World->GetTimerManager().SetTimer(Handle, TimerCallback, 0.5f, false);

   LambdaServer = FLambdaRunnable::RunLambdaOnBackGroundThread([&]()
   {
      // Build an audio stream to play sound data as it is received through the network
      Disconnecter Disconnect("localhost", 2435);
      Disconnect.start();
      while (WaitLittle);
   });

Title: Re: VoIp System where server passes clients speech to every other client.
Post by: Deliciouscookie on July 07, 2018, 11:13:51 pm

Quote from: Deliciouscookie on July 03, 2018, 08:42:55 pm

Quote from: Deliciouscookie on July 03, 2018, 07:49:41 pm

Quote from: eXpl0it3r on July 03, 2018, 05:41:53 pm

You're using blocking sockets, so the receive calls will block until they receive data. Which also means that your function gets stuck waiting for data.

Maybe there are more issues that I missed by glancing over it.

I think I found out the culprit. It is because stop() function won't work if the EndStream data won't come over network. I tested this by doing this:

Quote

                        if (id == endOfStream)
                        {
                           // End of stream reached: we stop receiving audio data
                           //   std::cout << "Audio data has been 100% received!" << std::endl;
                           stop();
                           Canrun = false;
                           UE_LOG(LogTemp, Warning, TEXT("Audio Stream successfully finished"));
                        }

If I create server and connect client through other program and then send endOfStream id, program goes pass the stop() and I get the LOG printing. If I create client and connect it through the visual studio code and then send endOfStream id, I never get the LOG printing, so program halts to stop and never goes pass it. Also, If I remove all the code and not run the stop(), the program never ends. The problem was never that I won't getting out of the loop. Problem is that I cannot run the stop() command for some reason.

So, question is, why the stop() won't run for me.

Got it working. Problem was when I spawned the class that sole purpose was to send the endStream message, it was destroyed too soon. So I added a little timer so the buffer get some data and then the endStream message goes through and everything closes nicely. Not best way i'm sure, but atleast now it's working without constant crashes

Quote

   FTimerDelegate TimerCallback;
   TimerCallback.BindLambda([&]
   {
      WaitLittle = false;
   });

   FTimerHandle Handle;
   World->GetTimerManager().SetTimer(Handle, TimerCallback, 0.5f, false);

   LambdaServer = FLambdaRunnable::RunLambdaOnBackGroundThread([&]()
   {
      // Build an audio stream to play sound data as it is received through the network
      Disconnecter Disconnect("localhost", 2435);
      Disconnect.start();
      while (WaitLittle);
   });

I come here again, asking for your guidance. This time, I created a video that demonstrates the problem I am having. The way I previously did the disconnecting(creating new client, connect it to server and send EndofStream) is too hacky way to do it, so I rather won't do it like that. If you have time to watch the video and give your thoughts about it, I would be very grateful. If you have any questions about the video, please let me know.

Here is the youtube link to video:
https://www.youtube.com/watch?v=u2_i-V8lfhg

edit: My guess is that it has something to do with the background thread running in SFML that won't shutdown properly.