SFNUL - Simple and Fast Network Utility Library
I originally intended it to just be called NUL, but go figure why I couldn't name it that... I guess the SF is fitting because if you ask me, it sticks with the SF philosophy somewhat :).
Background History
SFNUL was envisioned a veeeery long time ago, while I was still actively working on my super awesome project Wyrm (https://github.com/binary1248/Wyrm) before even touching SFGUI because Wyrm needed a UI :). Back then I already considered making the networking more reusable because I tended to have to rewrite the exact same code in every networked game I coded, and it got quite annoying. Time passed, and I had more interesting things to do (http://en.sfml-dev.org/forums/index.php?topic=6112.0) than actively work on Wyrm so that idea kind of went with it for a while.
It wasn't until helping Jungletoe debug his network related problems earlier this year that I realized SFML networking had a lot of shortcomings that I also found annoying, not only because I found that it didn't provide a level of abstraction of networking that was high enough, but because in certain situations it sacrifices standards conformance for a bit of usability. Being a computer science student for so long, I've come to revere standards which is why SFNUL aims to warn the developer about non-standard behaviour that they might cause.
General Idea
The main goal of SFNUL is to cut down on the network boilerplate code that can be seen in most multiplayer games. I already experienced this first hand while working on Wyrm, and browsing through the SFML network forum reviewing code and problems many have, I've gotten a feeling for what multiplayer game developers might need to spend less time writing "the same old annoying code that has to be written in every single project". SFNUL's sole purpose is to give developers more time to work on the code that makes their project unique from all others instead of code that is common among all projects. I want to see less code/problems and more playable multiplayer games ;).
Features
Fully asynchronous socket API.
Process data according to your own pace, it will trickle into an internal buffer but this doesn't mean you need to dedicate a thread to a blocking call or resort to selectors to check if any data is available. If data is available, you will be able to retrieve it and process it, if not then do something else.
Fully asynchronous TCP connection establishment/accept.
TCP connections will continue trying to connect and will automatically transmit/receive queued data when it is established. Incoming TCP connection requests are also queued into an internal buffer for you to process when you are ready.
TLS/SSL support.
You can set up a TLS transport and use it like you would use a TCP connection. All TLS specific features are added on top of the ReliableTransport interface which a normal TCP socket also implements.
HTTP/HTTPS client support.
Send arbitrarily constructed HTTP requests through the HTTPClient and it will receive and parse the reply sent back from the server which you can pick up from the HTTPClient, as with everything else, when you feel like it. HTTPClient supports HTTP/1.1 chunked transfers, persistent connections and request pipelining for efficient throughput utilization.
sfn::Message
Those familiar with sf::Packet will understand how sfn::Message works. It is the SFNUL version of sf::Packet that can do a lot more useful things ;). Let's use some code to demonstrate the differences:
sf::Packet packet;
sfn::Message message;
std::vector<int> vec = { 1, 2, 3, 4 };
packet << vec; // Doesn't compile, no operator<<( const std::vector<int>& )
message << vec; // Compiles, all objects that conform to the STL container interface are supported.
// Nesting containers works as well.
std::vector<std::list<std::deque<std::string>>> something;
// Insert some data here...
message << something; // Works fine.
message >> something; // Works fine as well.
int i = 42;
packet.prepend( &i, sizeof( i ) ); // Doesn't exist. No easy way to prepend data to a packet.
i >> message; // Intuitively insert data into the front of the message to prepend.
struct A {
int a, b, c;
float d, e, f;
char g[16];
};
A a;
packet << a; // What?? How could this possibly be implemented?
message << a; // Yes... this works... any trivial data type is supported for insertion and extraction.
message >> a; // Just to complete the cycle.
Links
Did you ever wonder how you can separate multiple different networking concerns from each other when communicating between hosts? Say you need to transmit data every time the player hits a key to move, that is one concern, then you need to transmit object states, that is another concern, and on top of that the players want to chat with each other, yet another concern. A typical approach would be to label each of these with a separate packet type ID to properly identify their payload, or the worse solution that many big games go for: open up multiple connections between the same pair of hosts. With the sfn::Link, that is taken care of automatically. sfn::Link multiplexes multiple distinct streams into a single network connection and demultiplexes them at the other end. All you need to do is assign each concern a stream ID and send/receive over that stream when dealing with that concern.
Object Synchronization
The centrepiece of SFNUL, seeking to solve the most annoying part of multiplayer coding. When you have multiple hosts which share the same world state with each other, performing the proper synchronization between them can become quite a tedious task. You need to assign IDs, perform required serialization/deserialization, object lifetime management, connection management and many other annoying things that have nothing to do with your vision of the game. This is where the sfn::Synchronizer comes in. Simply put: The sfn::Synchronizer makes sure that state changes to objects that it manages are propagated to synchronized remote hosts automatically without any extra coding effort from the developer. A simple example with a lot of things omitted for demonstration purposes:
class Blob : public sfn::SyncedObject {
public:
// ... type id / constructors omitted ...
sfn::SyncedInt32 x;
sfn::SyncedInt32 y;
sfn::SyncedType<sf::Color> c;
};
sfn::SynchronizerServer synchronizer_server;
auto blob = synchronizer.CreateObject<Blob>();
sfn::SynchronizerClient synchronizer_client;
// We assume the blob is already created on the client for brevity sake.
// On the server we do:
blob.x = 10;
blob.y = 20;
blob.c = sf::Color::Green;
// On the client we do.... absolutely nothing. The changes are performed automatically.
No more wasted time on getting the synchronization up and running and even more wasted time debugging it when it doesn't work. Do what you would do for a single player game and get multiplayer for free... in some sense :).
Here is a diagram showing how the SFNUL facilities and dependencies can interact with each other:
(http://i.imgur.com/yzgfTkG.png)
These were very brief examples/descriptions of the ideas behind SFNUL. If you want more demonstrations (that also work) look in the examples directory.
OK, enough with the features, where's the...
Yes... the repository can be found at GitHub: https://github.com/binary1248/SFNUL
If it isn't already apparent, SFNUL makes very heavy use of C++11 features. This means that you not only need a compiler that can compile C++11 code somewhat, but one that is feature complete and more importantly conforms to the standard. This already rules out Visual Studio 2012 and earlier, so if you haven't hopped on the Visual Studio 2013 train, well this could be another reason to consider if you want to try SFNUL.
SFNUL is licensed under the Mozilla Public License Version 2.0. Put simply, it is a file-based license, as long as you don't modify any of SFNUL's files you are free to do/distribute whatever/however you want, all subject to the terms of the MPL of course. If you distribute software using modified MPL code, you need to make available the modifications under the MPL as well. See this as a contribution back to the open source community for all of its efforts you have taken advantage of in the past ;). As with any other license, read through the MPL text if you want to understand what you can/cannot/have to do.
Frequently Asked Questions
Q: Where are the pre-compiled libraries for XYZ compiler?
A: Learn to compile something yourself, it isn't as hard as you think.
Q: I downloaded the zip archive off GitHub but it's complaining about missing headers during compile. Why?
A: zip archive won't work. Clone via git and initialize all submodules. Explained in the README.
Q: But I don't have git, surely there must be another way?
A: There is another way, it's called install git and use it.
Q: Wait... so what has this got to do with SFML?
A: Initially, it interacted a lot with SFML's classes, but then I realized that it would be easier for me to implement my own take on SFML's classes with many more features I found were missing and were feasible with C++11. So right now, it has very little to do with SFML. It can even be used independently of SFML. But you will recognize a bit of SFML's interface design still left in SFNUL.
Q: Do you plan on ....
A: Yes I have plans for a lot of things, and they will be implemented when I have a general idea of how to implement them. Either that, or I run into a dead end in one of my other projects and know that it is something SFNUL should be able to solve, so I implement it into SFNUL to be able to use it in the project and share it with the world at the same time.
Q: I know this isn't a question but I found a bug and I am sure it isn't intended behaviour described in a footnote of the documentation.
A: GitHub tracker.
Q: Where is the documentation?
A: doxygen inside the doc directory, or just generate the doc target in CMake.
Q: This is too complicated for me. Is there a simpler way?
A: What? This is supposed to make your life simpler. Write a few games with decent network support and come back here so that you can realize how that question doesn't make much sense.
Q: Are those the only questions in the FAQ?
A: Nope, more will be added if needed.
Ok. I feel alot of anger against microsoft i think. But its ok. I hate their attitude too. So much nice technology they have but always bound on their systems.
This has nothing to do with anger, the only think you might read from in between the lines is frustration. It's a fact that Microsoft didn't really try to catch up with the latest standard (C++11) and unfortunately they can just do that, since Microsoft made many companies and private developers depend on VS, that way they don't have to fear any drop in user count/sales.
They were visionary with VS 2010, because that version had already a few features (e.g. smart pointers) before the new standard was finalized, but with VS 2012 they barely even tried to catch up and even with VS 2013 they rather started to implement C++14 features instead of pushing out a fully C++11 conformant compiler.
The other fact is, that VS costs a ton of money if you don't want the Express version.
On the other side, you have the two open source compilers GCC, which essentially has everything implemented (http://gcc.gnu.org/projects/cxx0x.html) for month (http://gcc.gnu.org/ml/gcc-announce/2013/msg00004.html), and Clang, which - now hold your breath - is already C++14 feature complete (http://clang.llvm.org/cxx_status.html)!
So the frustration is really not just based on nothing - you really have to think about, how a company as big as Microsoft is and the money they have, doesn't manage to push out a properly working compiler in time. And not to forget people actually pay a lot for such unfinished work.
Error 2 error C2610: 'sfn::Thread::Thread(sfn::Thread &&)' : is not a special member function which can be defaulted C:\SFNUL\Concurrency.hpp 28 1 sfnul
Although this should've been fixed with the VS 2013 November CTP (http://blogs.msdn.com/b/vcblog/archive/2013/11/18/announcing-the-visual-c-compiler-november-2013-ctp.aspx) it seems like the toolchain is still bugged (http://herbsutter.com/2013/11/18/visual-c-compiler-november-2013-ctp/#comment-13312).
If you want to make it work, you can change stuff around on your own. The official source however won't get changed due to some compiler no conforming to the standard.
I dont know if i understand. You say you have to rewrite all of the code for VS2012 that it runs under the native compiler Toolset v110, because the library uses C++11 code. v110 in VS2012 is C++11 too. Perhaps not has all features of it.
Visual Studio 2012 only supports some of the features C++11 (http://msdn.microsoft.com/en-us/library/hh567368.aspx) defines in the standard. SFNUL however makes use of features that are not implemented for VS 2012. So regardless whether VS 2012 supports some features, it's still not a fully conforming C++11 compiler and it can't compile SFNUL ever - not with some simple tweak, not with some ugly hack, not with ..., it just won't compile with VS 2012 and you should deal with that.
I see too there are a lot of dependencies like boost too -> # include <boost/array.hpp> or <boost/weak_ptr.hpp> <boost/shared_ptr.hpp> which are not included in the project. Most people like me dont like boost because of the overhead you get of using parts of boost.
I see that the boost depencency comes from ASIO and they have a newer build that works without Boost and fully with C++11 features called Asio Standalone: http://think-async.com/Asio/AsioStandalone
It would be great if the awesome binary1248 update his library to the latest version to run it without boost.
No idea what you're looking at, but SFNUL is already based on a standalone version of ASIO, boost is not required.
Summary
- You can't use SFNUL with VS 2012.
- You might be able to use SFNUL with VS 2013, by applying your own modifications.
- What you really want, is to move away from VS and start using standard conforming compilers.
- This library is for advanced developers, you should know your tool chains good and should understand basic concepts of library comparability.
I can not believe that such a briliant piece of code is not buildable in windows. I dont want to believe it.
It is possible to build SFNUL using MSVC 2013, I just did it myself. Here is what it took for me to get it to build.
- Add "BOTAN_DLL=" to the preprocessor
- Add "NOEXCEPT=" to the preprocessor
- Add "__func__=__FUNCTION__" to the preprocessor
- Replace all instances of "noexcept" keyword with "NOEXCEPT" including in the external libraries
- Add default constructors to the Botan::HTTP::Response and Botan::OCSP::Response classes
Also change the code in tls_channel.h
from:
std::map<u16bit, std::shared_ptr<Connection_Cipher_State>> m_write_cipher_states =
{ { 0, nullptr } };
std::map<u16bit, std::shared_ptr<Connection_Cipher_State>> m_read_cipher_states =
{ { 0, nullptr } };
to:
std::map<u16bit, std::shared_ptr<Connection_Cipher_State>> m_write_cipher_states;
std::map<u16bit, std::shared_ptr<Connection_Cipher_State>> m_read_cipher_states;
and the code in tls_channel.cpp
from:
Channel::Channel(std::function<void (const byte[], size_t)> output_fn,
std::function<void (const byte[], size_t)> data_cb,
std::function<void (Alert, const byte[], size_t)> alert_cb,
std::function<bool (const Session&)> handshake_cb,
Session_Manager& session_manager,
RandomNumberGenerator& rng,
size_t reserved_io_buffer_size) :
m_handshake_cb(handshake_cb),
m_data_cb(data_cb),
m_alert_cb(alert_cb),
m_output_fn(output_fn),
m_rng(rng),
m_session_manager(session_manager)
{
m_writebuf.reserve(reserved_io_buffer_size);
m_readbuf.reserve(reserved_io_buffer_size);
}
to:
Channel::Channel(std::function<void (const byte[], size_t)> output_fn,
std::function<void (const byte[], size_t)> data_cb,
std::function<void (Alert, const byte[], size_t)> alert_cb,
std::function<bool (const Session&)> handshake_cb,
Session_Manager& session_manager,
RandomNumberGenerator& rng,
size_t reserved_io_buffer_size) :
m_handshake_cb(handshake_cb),
m_data_cb(data_cb),
m_alert_cb(alert_cb),
m_output_fn(output_fn),
m_rng(rng),
m_session_manager(session_manager),
m_write_cipher_states(),
m_read_cipher_states()
{
m_write_cipher_states.insert({ 0, nullptr });
m_read_cipher_states.insert({ 0, nullptr });
m_writebuf.reserve(reserved_io_buffer_size);
m_readbuf.reserve(reserved_io_buffer_size);
}
So ive implemented this library for a prototype to test the features i need to making a working solution for my game.
At the moment i see a problem when try to remove a synchronized object that is not the last object that was pushed to the list. Its not working proper.
There is no way to use pointers for the synchronizer, so i have to carry around values.
When using _synchronizerServer.CreateObject<Entity>();
Also, the copy constructor will be called 2 times. With a pointer it shouldnt be such a cost problem.
But the main problem is that Synchroned Objects are values which i get over the CreateObject-Method where i cant really make a reference from, beside the list where the synchronized objects stored.
The memory allocation is changing automaticly when objects are removed form the STL containers, i guess.
I cant see how the STL container-classes manage the allocation of memory, but i see that something is wrong when i try to make something like:
_entities.push_back(_synchronizerServer->CreateObject<Entity>());
Entity& entityA = _entities.back();
So when i try to use entityA later, after i removed some other objects out of the _entities-list, it will shows that the reference is not valid anymore.
I think without modifying the synchronizer it wont work. I need pointers. Interrupt me, when im wrong, but is see no other way for the problem.
So ive manipulated the synchronizer.inl file like this and use now _synchronizerServer->CreateObject2<Entity>()
template<typename T, typename... Args>
T* SynchronizerServer::CreateObject2(Args&&... args) {
T* object = new T(std::forward<Args>(args)...);
object->SetSynchronizer(this);
return object;
}
And now it works for me like it should.
I see two architecture hurdles,
A) Synchronization of sub-syncObjects of a syncObject.
B) Different views. At the moment i see that all syncObjects are there for synchronize all clients with the same objects, but why should objects be synchronized with clients/players where the object is currently dont visible. So i need more than one synchronizer (for each client (player)) that only synchronize the visible objects for a client.
So at the end it cant be solved in the way that the synchronizer handles the creation of the SyncObjects like it do with the CreateObject-Method.
With this way there is for each synchronizer a represenation of a SyncObject on Server-Side. So i have to remove the CreateObject-Method and give the synchronizer directly the SyncObject to make it possible to share the syncObject with all synchronizer (for each client) on the server.
Can you compile this?
#include <mutex>
int main()
{
std::mutex m;
}
Can you compile this?
#include <mutex>
int main()
{
std::mutex m;
}
It isn't working like this either. I wonder why std::mutex it's not being included with MinGW.
For the lazy people here's a simple batch file that will build SFNUL as static libs in debug and release mode.
Prerequisites:
- Git for Windows (https://msysgit.github.io/) must be found in PATH
- CMake (http://www.cmake.org/download/) must be found in PATH
- MinGW (http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/mingw-builds/4.9.2/threads-posix/dwarf/) must be found in PATH
git clone https://github.com/binary1248/SFNUL.git
cd SFNUL
git submodule update --init --recursive
mkdir build
cd build
cmake -G"MinGW Makefiles" -DSFNUL_USE_STD_THREAD=TRUE -DCMAKE_INSTALL_PREFIX=../install -DCMAKE_BUILD_TYPE=Debug -DSFNUL_BUILD_EXAMPLES=FALSE ..\
mingw32-make install -j8
cmake -G"MinGW Makefiles" -DSFNUL_USE_STD_THREAD=TRUE -DCMAKE_INSTALL_PREFIX=../install -DCMAKE_BUILD_TYPE=Release -DSFNUL_BUILD_EXAMPLES=FALSE ..\
mingw32-make install -j8
The builds can finally be found in the SFNUL/install directory.