Welcome, Guest. Please login or register. Did you miss your activation email?

Author Topic: SFNUL  (Read 51381 times)

0 Members and 1 Guest are viewing this topic.

binary1248

  • SFML Team
  • Hero Member
  • *****
  • Posts: 1405
  • I am awesome.
    • View Profile
    • The server that really shouldn't be running
SFNUL
« on: November 30, 2013, 04:32:03 am »
SFNUL - Simple and Fast Network Utility Library

I originally intended it to just be called NUL, but go figure why I couldn't name it that... I guess the SF is fitting because if you ask me, it sticks with the SF philosophy somewhat :).

Background History
SFNUL was envisioned a veeeery long time ago, while I was still actively working on my super awesome project Wyrm before even touching SFGUI because Wyrm needed a UI :). Back then I already considered making the networking more reusable because I tended to have to rewrite the exact same code in every networked game I coded, and it got quite annoying. Time passed, and I had more interesting things to do than actively work on Wyrm so that idea kind of went with it for a while.

It wasn't until helping Jungletoe debug his network related problems earlier this year that I realized SFML networking had a lot of shortcomings that I also found annoying, not only because I found that it didn't provide a level of abstraction of networking that was high enough, but because in certain situations it sacrifices standards conformance for a bit of usability. Being a computer science student for so long, I've come to revere standards which is why SFNUL aims to warn the developer about non-standard behaviour that they might cause.

General Idea
The main goal of SFNUL is to cut down on the network boilerplate code that can be seen in most multiplayer games. I already experienced this first hand while working on Wyrm, and browsing through the SFML network forum reviewing code and problems many have, I've gotten a feeling for what multiplayer game developers might need to spend less time writing "the same old annoying code that has to be written in every single project". SFNUL's sole purpose is to give developers more time to work on the code that makes their project unique from all others instead of code that is common among all projects. I want to see less code/problems and more playable multiplayer games ;).

Features
Fully asynchronous socket API.
Process data according to your own pace, it will trickle into an internal buffer but this doesn't mean you need to dedicate a thread to a blocking call or resort to selectors to check if any data is available. If data is available, you will be able to retrieve it and process it, if not then do something else.

Fully asynchronous TCP connection establishment/accept.
TCP connections will continue trying to connect and will automatically transmit/receive queued data when it is established. Incoming TCP connection requests are also queued into an internal buffer for you to process when you are ready.

TLS/SSL support.
You can set up a TLS transport and use it like you would use a TCP connection. All TLS specific features are added on top of the ReliableTransport interface which a normal TCP socket also implements.

HTTP/HTTPS client support.
Send arbitrarily constructed HTTP requests through the HTTPClient and it will receive and parse the reply sent back from the server which you can pick up from the HTTPClient, as with everything else, when you feel like it. HTTPClient supports HTTP/1.1 chunked transfers, persistent connections and request pipelining for efficient throughput utilization.

sfn::Message
Those familiar with sf::Packet will understand how sfn::Message works. It is the SFNUL version of sf::Packet that can do a lot more useful things ;). Let's use some code to demonstrate the differences:
sf::Packet packet;
sfn::Message message;

std::vector<int> vec = { 1, 2, 3, 4 };

packet << vec; // Doesn't compile, no operator<<( const std::vector<int>& )
message << vec; // Compiles, all objects that conform to the STL container interface are supported.

// Nesting containers works as well.
std::vector<std::list<std::deque<std::string>>> something;
// Insert some data here...

message << something; // Works fine.
message >> something; // Works fine as well.

int i = 42;

packet.prepend( &i, sizeof( i ) ); // Doesn't exist. No easy way to prepend data to a packet.
i >> message; // Intuitively insert data into the front of the message to prepend.

struct A {
        int a, b, c;
        float d, e, f;
        char g[16];
};

A a;

packet << a; // What?? How could this possibly be implemented?
message << a; // Yes... this works... any trivial data type is supported for insertion and extraction.
message >> a; // Just to complete the cycle.

Links
Did you ever wonder how you can separate multiple different networking concerns from each other when communicating between hosts? Say you need to transmit data every time the player hits a key to move, that is one concern, then you need to transmit object states, that is another concern, and on top of that the players want to chat with each other, yet another concern. A typical approach would be to label each of these with a separate packet type ID to properly identify their payload, or the worse solution that many big games go for: open up multiple connections between the same pair of hosts. With the sfn::Link, that is taken care of automatically. sfn::Link multiplexes multiple distinct streams into a single network connection and demultiplexes them at the other end. All you need to do is assign each concern a stream ID and send/receive over that stream when dealing with that concern.

Object Synchronization
The centrepiece of SFNUL, seeking to solve the most annoying part of multiplayer coding. When you have multiple hosts which share the same world state with each other, performing the proper synchronization between them can become quite a tedious task. You need to assign IDs, perform required serialization/deserialization, object lifetime management, connection management and many other annoying things that have nothing to do with your vision of the game. This is where the sfn::Synchronizer comes in. Simply put: The sfn::Synchronizer makes sure that state changes to objects that it manages are propagated to synchronized remote hosts automatically without any extra coding effort from the developer. A simple example with a lot of things omitted for demonstration purposes:
class Blob : public sfn::SyncedObject {
public:
        // ... type id / constructors omitted ...
        sfn::SyncedInt32 x;
        sfn::SyncedInt32 y;
        sfn::SyncedType<sf::Color> c;
};

sfn::SynchronizerServer synchronizer_server;
auto blob = synchronizer.CreateObject<Blob>();

sfn::SynchronizerClient synchronizer_client;
// We assume the blob is already created on the client for brevity sake.

// On the server we do:
blob.x = 10;
blob.y = 20;
blob.c = sf::Color::Green;

// On the client we do.... absolutely nothing. The changes are performed automatically.
No more wasted time on getting the synchronization up and running and even more wasted time debugging it when it doesn't work. Do what you would do for a single player game and get multiplayer for free... in some sense :).

Here is a diagram showing how the SFNUL facilities and dependencies can interact with each other:


These were very brief examples/descriptions of the ideas behind SFNUL. If you want more demonstrations (that also work) look in the examples directory.

OK, enough with the features, where's the...
Yes... the repository can be found at GitHub: https://github.com/binary1248/SFNUL

If it isn't already apparent, SFNUL makes very heavy use of C++11 features. This means that you not only need a compiler that can compile C++11 code somewhat, but one that is feature complete and more importantly conforms to the standard. This already rules out Visual Studio 2012 and earlier, so if you haven't hopped on the Visual Studio 2013 train, well this could be another reason to consider if you want to try SFNUL.

SFNUL is licensed under the Mozilla Public License Version 2.0. Put simply, it is a file-based license, as long as you don't modify any of SFNUL's files you are free to do/distribute whatever/however you want, all subject to the terms of the MPL of course. If you distribute software using modified MPL code, you need to make available the modifications under the MPL as well. See this as a contribution back to the open source community for all of its efforts you have taken advantage of in the past ;). As with any other license, read through the MPL text if you want to understand what you can/cannot/have to do.

Frequently Asked Questions
Q: Where are the pre-compiled libraries for XYZ compiler?
A: Learn to compile something yourself, it isn't as hard as you think.

Q: I downloaded the zip archive off GitHub but it's complaining about missing headers during compile. Why?
A: zip archive won't work. Clone via git and initialize all submodules. Explained in the README.

Q: But I don't have git, surely there must be another way?
A: There is another way, it's called install git and use it.

Q: Wait... so what has this got to do with SFML?
A: Initially, it interacted a lot with SFML's classes, but then I realized that it would be easier for me to implement my own take on SFML's classes with many more features I found were missing and were feasible with C++11. So right now, it has very little to do with SFML. It can even be used independently of SFML. But you will recognize a bit of SFML's interface design still left in SFNUL.

Q: Do you plan on ....
A: Yes I have plans for a lot of things, and they will be implemented when I have a general idea of how to implement them. Either that, or I run into a dead end in one of my other projects and know that it is something SFNUL should be able to solve, so I implement it into SFNUL to be able to use it in the project and share it with the world at the same time.

Q: I know this isn't a question but I found a bug and I am sure it isn't intended behaviour described in a footnote of the documentation.
A: GitHub tracker.

Q: Where is the documentation?
A: doxygen inside the doc directory, or just generate the doc target in CMake.

Q: This is too complicated for me. Is there a simpler way?
A: What? This is supposed to make your life simpler. Write a few games with decent network support and come back here so that you can realize how that question doesn't make much sense.

Q: Are those the only questions in the FAQ?
A: Nope, more will be added if needed.
« Last Edit: November 30, 2013, 12:42:16 pm by binary1248 »
SFGUI # SFNUL # GLS # Wyrm <- Why do I waste my time on such a useless project? Because I am awesome (first meaning).

eXpl0it3r

  • SFML Team
  • Hero Member
  • *****
  • Posts: 10800
    • View Profile
    • development blog
    • Email
AW: SFNUL
« Reply #1 on: November 30, 2013, 11:06:26 am »
Everyone is so overwhelmed with your awesomeness, that they don't manage to say anything. :D

As a tester and to some extend user of SFNUL I can only recommend this library. I've never really done any network programming, but with SFNUL it indeed is very easy to achieve what you want, without having to worry about "low level" stuff.
I certainly will make further use of SFNUL whenever possible. ;)
Official FAQ: https://www.sfml-dev.org/faq.php
Official Discord Server: https://discord.gg/nr4X7Fh
——————————————————————
Dev Blog: https://duerrenberger.dev/blog/

Nexus

  • SFML Team
  • Hero Member
  • *****
  • Posts: 6286
  • Thor Developer
    • View Profile
    • Bromeon
Re: SFNUL
« Reply #2 on: November 30, 2013, 11:45:13 am »
Definitely a very interesting project!

My networking experience with SFML is limited, but I can confirm that it takes quite some time to get things right. A library that simplifies that would definitely be very helpful. Also, it's amazing to see secure connections over SSL/TLS. It was probably not too easy to implement that with certificates and everything (I don't know how much the underlying libraries already do here)...
Zloxx II: action platformer
Thor Library: particle systems, animations, dot products, ...
SFML Game Development:

binary1248

  • SFML Team
  • Hero Member
  • *****
  • Posts: 1405
  • I am awesome.
    • View Profile
    • The server that really shouldn't be running
Re: SFNUL
« Reply #3 on: November 30, 2013, 12:43:14 pm »
It was probably not too easy to implement that with certificates and everything (I don't know how much the underlying libraries already do here)...
SFNUL uses TropicSSL for its TLS/SSL layer on top of a normal TCP socket. This means that the transport layer is still taken care of by SFNUL which itself relies on asio instead of letting TropicSSL resort to its standard POSIX socket calls. TropicSSL is the last revision of what is now named PolarSSL that was licensed under the BSD license, but if you consider that no big advancements have been made in the crypto-scene in the last few years, there isn't much of a difference.

SFNUL was designed to be built up in layers (see diagram I added to the original post). Mixing multiple layers together in different combinations to achieve what you need easily is the goal. At the moment most combinations (including exotic ones) work, but a few more obvious ones don't work due to technical reasons. This might change in the future.
SFGUI # SFNUL # GLS # Wyrm <- Why do I waste my time on such a useless project? Because I am awesome (first meaning).

Laurent

  • Administrator
  • Hero Member
  • *****
  • Posts: 32504
    • View Profile
    • SFML's website
    • Email
Re: SFNUL
« Reply #4 on: November 30, 2013, 06:42:11 pm »
This looks awesome. I'm sure I'll find many interesting ideas in it for sfml-network 3.0 :)
Laurent Gomila - SFML developer

Lo-X

  • Hero Member
  • *****
  • Posts: 618
    • View Profile
    • My personal website, with CV, portfolio and projects
Re: SFNUL
« Reply #5 on: December 02, 2013, 11:18:41 am »
I'm speechless :o

I just get myself in a networking game and as for now I think I'm going to use your framework.

The SyncedObject thing look awesome, I didn't even imagine such a thing. More snippets/examples may help (especially client-side, since you show only server-side).
The message version of packets will be sooo usefull as well, even if we could do it before with (some) extra code, definitely will make life simple !

A.W.E.S.O.M.E

Edit : why did you used CamelCase instead of camelCase for methods :( ?
« Last Edit: December 02, 2013, 01:48:55 pm by Lo-X »

binary1248

  • SFML Team
  • Hero Member
  • *****
  • Posts: 1405
  • I am awesome.
    • View Profile
    • The server that really shouldn't be running
Re: SFNUL
« Reply #6 on: December 02, 2013, 03:30:11 pm »
The SyncedObject thing look awesome, I didn't even imagine such a thing. More snippets/examples may help (especially client-side, since you show only server-side).
I kept it simple in the snippet so that people can understand the general idea without having to read 10 pages of code. A fully functional example of both client and server sides can be found with the source code in the examples directory.

Edit : why did you used CamelCase instead of camelCase for methods :( ?
I was just used to it ;D. Would you ask this question if SFML still used CamelCase? Maybe it's a good thing that different libraries use different naming conventions, easier to distinguish where an object comes from :).
SFGUI # SFNUL # GLS # Wyrm <- Why do I waste my time on such a useless project? Because I am awesome (first meaning).

Lo-X

  • Hero Member
  • *****
  • Posts: 618
    • View Profile
    • My personal website, with CV, portfolio and projects
Re: SFNUL
« Reply #7 on: December 02, 2013, 03:45:34 pm »
The SyncedObject thing look awesome, I didn't even imagine such a thing. More snippets/examples may help (especially client-side, since you show only server-side).
I kept it simple in the snippet so that people can understand the general idea without having to read 10 pages of code. A fully functional example of both client and server sides can be found with the source code in the examples directory.

Yeah, I saw that afterward, I've what I needed (for now :p )

Edit : why did you used CamelCase instead of camelCase for methods :( ?
I was just used to it ;D. Would you ask this question if SFML still used CamelCase? Maybe it's a good thing that different libraries use different naming conventions, easier to distinguish where an object comes from :).

I'm used to camelCase, I used to wonder why SFML used CamelCase back in 1.6.
You're right and wrong for the origin of objects. If we only use your lib (or very few) we can know it. But as soon as you use multiple libs, it mixed up between camel and Camel.
I said that because I like my code to be "nice" (in the aspect way), consistent. And I'm used to STL and/or Boost and/or Qt (and now SFML) and their camelCase convention.

It wasn't a request, I just wanted to know if there was any reason.
It doesn't withdraw any quality or amazing-ness from your lib :p I'll test it further this evening.

binary1248

  • SFML Team
  • Hero Member
  • *****
  • Posts: 1405
  • I am awesome.
    • View Profile
    • The server that really shouldn't be running
Re: SFNUL
« Reply #8 on: December 02, 2013, 04:17:58 pm »
Doesn't STL use cpp_case? I was actually considering using cpp_case but I figured I don't want to step on the hallowed ground of STL/boost and scare people away :). That and I would get confused between functions and my own variables which already use cpp_case :P.
SFGUI # SFNUL # GLS # Wyrm <- Why do I waste my time on such a useless project? Because I am awesome (first meaning).

Lo-X

  • Hero Member
  • *****
  • Posts: 618
    • View Profile
    • My personal website, with CV, portfolio and projects
Re: SFNUL
« Reply #9 on: December 02, 2013, 05:13:58 pm »
Doesn't STL use cpp_case? I was actually considering using cpp_case but I figured I don't want to step on the hallowed ground of STL/boost and scare people away :). That and I would get confused between functions and my own variables which already use cpp_case :P.

Yes you're right, I was too fast and I dunno how I could forget how much I dislike cpp_case (it's indeed scary).
That's surely because I have basics methods in mind, like size() empty() stuff()

Is it possible to take just a part of your lib ? Essentially I don't need all the SSL stuff (for now) but I would like to use the "basic" network part, for games. I would like to adapt it to my code and learn how you did all that, so I'm rather thinking about re-(copying)-coding it if it's possible (with credits where it's due, of course) ?

Edit: it could sove my very big issue with camelCase ;p

binary1248

  • SFML Team
  • Hero Member
  • *****
  • Posts: 1405
  • I am awesome.
    • View Profile
    • The server that really shouldn't be running
Re: SFNUL
« Reply #10 on: December 02, 2013, 06:59:06 pm »
If you release the networking code under the MPL as well, I don't see why you couldn't take parts of it and modify it for your own purposes... The TLS functionality is coupled to the HTTP client so if you don't want to have it, you have to remove HTTP support as well. But I don't see why it would be a problem to just leave TLS support in the library. What ends up in your final executable is what you actually use (if linking statically). If you want to thin down the library for dynamic linking then yes, you will have to manually get rid of parts yourself.
SFGUI # SFNUL # GLS # Wyrm <- Why do I waste my time on such a useless project? Because I am awesome (first meaning).

Lo-X

  • Hero Member
  • *****
  • Posts: 618
    • View Profile
    • My personal website, with CV, portfolio and projects
Re: SFNUL
« Reply #11 on: December 02, 2013, 07:18:13 pm »
If you release the networking code under the MPL as well, I don't see why you couldn't take parts of it and modify it for your own purposes... The TLS functionality is coupled to the HTTP client so if you don't want to have it, you have to remove HTTP support as well. But I don't see why it would be a problem to just leave TLS support in the library. What ends up in your final executable is what you actually use (if linking statically). If you want to thin down the library for dynamic linking then yes, you will have to manually get rid of parts yourself.

Yeah no that's not a question about how much of the lib I want in my executable, I don't care :p That's about how you did things, perhaps why and what I would have done differently. Learning purposes, since I did some (few) networking for 2/3 games, I'm curious about your SyncedObjects and stuff like that

TypeOverride

  • Newbie
  • *
  • Posts: 14
    • View Profile
    • Email
Re: SFNUL
« Reply #12 on: December 21, 2013, 10:34:50 pm »
Ive try to run the library under Visual Studio 2012.

It cant be run because tropicssl doesnt work with Visual Studio 2012, because Visual Studio 2012 has no <inttypes>
So ive used http://code.google.com/p/msinttypes/ for the inttypes.

But there are enough other errors under Visual Studio 2012 that makes the library kind of hard to build.

zsbzsb

  • Hero Member
  • *****
  • Posts: 1409
  • Active Maintainer of CSFML/SFML.NET
    • View Profile
    • My little corner...
    • Email
Re: SFNUL
« Reply #13 on: December 21, 2013, 11:47:02 pm »
Ive try to run the library under Visual Studio 2012.

It cant be run because tropicssl doesnt work with Visual Studio 2012, because Visual Studio 2012 has no <inttypes>
So ive used http://code.google.com/p/msinttypes/ for the inttypes.

But there are enough other errors under Visual Studio 2012 that makes the library kind of hard to build.

Don't you even read the original post?  ???

Quote
If it isn't already apparent, SFNUL makes very heavy use of C++11 features. This means that you not only need a compiler that can compile C++11 code somewhat, but one that is feature complete and more importantly conforms to the standard. This already rules out Visual Studio 2012 and earlier, so if you haven't hopped on the Visual Studio 2013 train, well this could be another reason to consider if you want to try SFNUL.
Motion / MotionNET - Complete video / audio playback for SFML / SFML.NET

NetEXT - An SFML.NET Extension Library based on Thor

TypeOverride

  • Newbie
  • *
  • Posts: 14
    • View Profile
    • Email
Re: SFNUL
« Reply #14 on: December 21, 2013, 11:48:28 pm »
Yes i read it. I try to make it runnable in vs2012

One other question is how others compile this library when he say that vs2012 cant be used. And a compiler is not enough.

Quote
This means that you not only need a compiler that can compile C++11
i dont understand what he means. What is it what i need beside a C++11 compiler that uses C++11 features heavily?

What i did was:
1) Using GIT loading the project (i recognize that the extlibs are not downloaded), so
2) ive GIT loaded all the extlibs too.
3) used CMake an making a VS2012 project
4) fixed the problem with tropicssl and the missing inttypes header in vs2012
5) then i saw the heavy used C++11 features for which i have to use an other compiler i think  - didnt make it at the moment, because ...

... perhaps its easier... because of that i ask here how others like you would compile SFNUL?
« Last Edit: December 22, 2013, 12:08:17 am by TypeOverride »

 

anything