Benchmark : SDL vs SFML

[Laurent]

Correct, I fixed it in a new version that I didn't upload.

Too bad SDL requires so much setup just to get things right

[Laurent]

Since this benchmark is still being criticized, I updated it to try making it as correct as possible.

The URL is still the same:
http://www.sfml-dev.org/temp/bench-sdl-sfml.zip

[jmpeer]

Your benchmarks are severly flawed. You are comparing the software, 2D library that ships with SDL to OpenGL. Of course you are going to get a huge difference in performance - you don't even need to perform a benchmark to understand that.

What would be really interesting is to see how an SDL-driven OpenGL application performs compared to SFML. That would be a meaningful test, for two reasons: it compares similar technologies and it benchmarks against SDL the way SDL is almost always used.

After seeing SDL_DisplayFormat optimize its performance, I'm worried about these benchmarks. I support the development of SFML, but you should measure the true performance by fully optimizing both sides like this guy mentioned.

[Laurent]

No, I really want to compare SFML to the 2D API of SDL. Comparing SFML to OpenGL would be ridiculous, SFML is built on top of OpenGL and they have a very different purpose.

The graphics modules of SDL and SFML are supposed to be similar aren't they? So why shouldn't I compare them? Because SDL is too slow and unusable? Well, if I don't do this kind of benchmark, who will ever notice it?

Basically everyone here is just saying that SDL doesn't have a decent 2D module, and that's exactly one of the things I wanted to show (the other being the simplicity of the code -- if you want me to compare to OpenGL it will be even worse for SDL!).

[jmpeer]

Ah, I got you.
Seems pretty cool anyways, I'd like to learn it.

[nitram_cero]

SFML's interface API got me hooked away from SDL  :lol:

Non-OOP game ("media") libraries are a pain in the balls because you probably end up wrapping it in classes that are just like SFML  

Even thou I must accept is a whole lot easier and faster to work on pixels in software mode. Reloading a whole texture for just a couple of pixels each frame is slower I believe.

But well, SDL was created a long time ago. It has gained it deserved popularity... but I think it's time to move on.

I mean, people are still using allegro. I used to read about allegro when I was a kid on game magazines... back then it was the shit (as in awesome).
From wikipedia:

Initially standing for "Atari Low-Level Game Routines"...for the Atari ST in the early 1990s.

ATARI, DUDE xD

Well, I think the best propaganda for developers is the one that compares numbers. Geez, If a politician showed me some numbers instead of sensless babling... he'd win my vote  :lol:

I will check the benchmark when I have some time.

Regards
-Martín

[Jonny D]

I took a stab at it on a 2nd gen Macbook running Debian.  There was only one build error: EXIT_SUCCESS is not defined in main.cpp.

I made a couple of changes and added two tests.  You only tested per-surface alpha-blending, so I added a test for per-pixel alpha (it uses SDL_image now).  You also made a couple of no-no's in the rotation and dynamic text tests by creating surfaces within the loop (allocating and freeing tons of memory per second).  I changed the rotation test to use SGE's direct rotations (and made it work correctly) and I plugged in NFont so that I could cache the font, rather than generate each glyph when needed.  These represent more truly the way that people actually use SDL and its supporting libs for speed.  So, Test 2 is not very indicative of typical usage, and Test 6 should be thrown right out.  One note: Test 7 uses only Normal font style, since NFont doesn't load the character width correctly for others.

You can get my version at http://code.bluedinosaurs.com/SDL/bench-sdl-sfml-j.zip

The results are not that amazing.  I'd like to see the results from other computers.  Also, if you mean this thread to tell beginners about the speed differences between SDL and SFML, then you need to be upfront about it.  The first results don't seem to be updated and you don't mention exactly what you're testing (software SDL vs hardware SFML) until a later page.  In the head post, it should state that SDL can be used to make hardware-accelerated graphics, but SDL's built-in 2d support, which is being tested, is in software, whereas SFML uses hardware acceleration.  Indeed, SDL 1.3 (as opposed to SDL 1.2 used in this benchmark) uses hardware acceleration on its 2d graphics, just like SFML.

My results:

1/ Test : sprites
SDL  displayed 47 frames
SFML displayed 51 frames
--> SFML is 1.09x as fast as SDL

2/ Test : alpha-blended sprites
SDL  displayed 38 frames
SFML displayed 52 frames
--> SFML is 1.37x as fast as SDL

3/ Test : per-pixel alpha-blended sprites
SDL  displayed 42 frames
SFML displayed 47 frames
--> SFML is 1.12x as fast as SDL

4/ Test : rotating sprites
SDL  displayed 26 frames
SFML displayed 26 frames
--> SFML is 1.00x as fast as SDL

5/ Test : static text
SDL  displayed 1382 frames
SFML displayed 1023 frames
--> SFML is 0.74x as fast as SDL

6/ Test : dynamic text
SDL  displayed 194 frames
SFML displayed 727 frames
--> SFML is 3.75x as fast as SDL

7/ Test : dynamic text w/NFont
SDL  displayed 779 frames
SFML displayed 1113 frames
--> SFML is 1.43x as fast as SDL

[Laurent]

I'm glad to see someone providing a really optimized SDL code, thanks.

First, this topic is not about the speed difference between SDL and SFML. It's about speed & easiness. The only thing that I want to show is that you can get both of them with SFML, not with SDL. You can get a code as simple as SFML (this is my first version of the benchmark) but with poor performances, or you can get a code almost as fast as SFML but with tons of addons and tweaks (this is your version).

That's all I want to show, and so far it still works

you don't mention exactly what you're testing (software SDL vs hardware SFML)

I'm testing SDL and SFML. Hardware / software is just an implementation detail.
Moreover, does hardware SDL exist? SDL 1.3 is still in progress, and SDL + OpenGL... well it's OpenGL, not SDL.

Indeed, SDL 1.3 (as opposed to SDL 1.2 used in this benchmark) uses hardware acceleration on its 2d graphics, just like SFML.

I already had a look at SDL 1.3, but was too lazy to try figuring out how it works (I mean, the public API). I'm really curious to see it in action

[Jonny D]

First, this topic is not about the speed difference between SDL and SFML. It's about speed & easiness. The only thing that I want to show is that you can get both of them with SFML, not with SDL.

Agreed
Us with a mind towards C++ do need to wrap SDL a bit to get comfortable.

You can get a code as simple as SFML (this is my first version of the benchmark) but with poor performances, or you can get a code almost as fast as SFML but with tons of addons and tweaks (this is your version).

I don't think this is quite true.  There is a big difference between SDL and SFML in that SDL covers the crossplatform basics and you choose libraries for your task (the "pick-and-choose" approach), whereas SFML wraps and makes available everything you might need (the "one-stop-shop" approach).  It's really up to the user to decide what is best for him/her.  For beginners, the latter approach is clearly better, in my opinion.  It is nice to have a consistent interface, but I prefer using several small libraries with maintainers who can optimize each one well.  I didn't tweak anything for the tests, but I did use the extension libraries the way they're meant to be used.  You are an expert with SFML, so I thought SDL needed someone with experience to make the benchmark more viable.

I'm testing SDL and SFML. Hardware / software is just an implementation detail.

It's not just an implementation detail.  It means real things on various platforms.  If you're on an embedded system with no GPU, then SDL and other software-optimized libraries are much, much faster than a software OpenGL implementation.

Moreover, does hardware SDL exist? SDL 1.3 is still in progress, and SDL + OpenGL... well it's OpenGL, not SDL.

Yes, hardware SDL exists.  In its original form, you've been using it by using SDL_HWSURFACE (i.e. requesting a surface in video memory).  In a form more comparable to SFML, you would either use glSDL or SDL 1.3.  glSDL is a replacement of the SDL backend with an OpenGL one.  A similar, but not identical, idea is what has become SDL 1.3.

I already had a look at SDL 1.3, but was too lazy to try figuring out how it works (I mean, the public API). I'm really curious to see it in action

Yeah, who knows when an official release will occur...  I look forward most to the multiple mice/keyboard/monitor/window stuff.

[Laurent]

There is a big difference between SDL and SFML in that SDL covers the crossplatform basics and you choose libraries for your task (the "pick-and-choose" approach), whereas SFML wraps and makes available everything you might need (the "one-stop-shop" approach)

This is not 100% true. SFML is divided into 5 different modules as well, and you can pick only what you need.
The thing with SDL is that the different modules are not consistent, there's no common base (unlike SFML). SDL, SDL_net, SDL_image, NFont, SGE, ... it looks like you need to gather a lot of different libraries, rather than using different modules of the same thing. It's confusing for beginners. And you don't easily find documentation for some modules (yesterday I was unable to find a reference for the sge_transform function that you use). Plus, I think that there are slightly too many modules; SDL_image is a good example: I think it is stupid to have to use a separate library just to be able to handle the most common image formats.

You are an expert with SFML, so I thought SDL needed someone with experience to make the benchmark more viable

Absolutely!

It's not just an implementation detail. It means real things on various platforms. If you're on an embedded system with no GPU, then SDL and other software-optimized libraries are much, much faster than a software OpenGL implementation.

But I'm not talking about such systems, SFML is only made for desktop PCs (for now). And people using a graphics library don't care about the internals of this library, they just want it to have good performances, whether it is hardware accelerated or not. Especially beginners, who know nothing about hardware / software and OpenGL stuff.

Yes, hardware SDL exists. In its original form, you've been using it by using SDL_HWSURFACE (i.e. requesting a surface in video memory)

So I'm indeed comparing hardware SDL to hardware SFML

glSDL is a replacement of the SDL backend with an OpenGL one

But it's not SDL. This is another example where you need another separate library to get something interesting from SDL.

A similar, but not identical, idea is what has become SDL 1.3.

I think I'll try to dive into the internals of SDL 1.3 and see how they render 2D stuff using OpenGL. It should be interesting

[richardwb]

I spent a bit of time and gathered the necessary libraries (it took some time) to provide an additional data point for this benchmark. I used Jonny D's version of the benchmark. This is on a Core 2 Quad @ 3.0GHz, 8GB RAM, Nvidia 8800GT under Vista 64-bit and Visual Studio 2008.

There were some visual inconsistencies in the dynamic text benchmark.

Code: [Select]

1/ Test : sprites
SDL  displayed 168 frames
SFML displayed 736 frames
--> SFML is 4.38x as fast as SDL

2/ Test : alpha-blended sprites
SDL  displayed 12 frames
SFML displayed 505 frames
--> SFML is 42.08x as fast as SDL

3/ Test : per-pixel alpha-blended sprites
SDL  displayed 16 frames
SFML displayed 709 frames
--> SFML is 44.31x as fast as SDL

4/ Test : rotating sprites
SDL  displayed 30 frames
SFML displayed 691 frames
--> SFML is 23.03x as fast as SDL

5/ Test : static text
SDL  displayed 756 frames
SFML displayed 7273 frames
--> SFML is 9.62x as fast as SDL

6/ Test : dynamic text
SDL  displayed 116 frames
SFML displayed 3697 frames
--> SFML is 31.87x as fast as SDL

7/ Test : dynamic text w/NFont
SDL  displayed 606 frames
SFML displayed 4575 frames
--> SFML is 7.55x as fast as SDL

[solgar]

Testing machine: some HP Pavilion dv7.
Result:

Code: [Select]

1/ Test : sprites
SDL  displayed 51 frames
SFML displayed 69 frames
--> SFML is 1.4x as fast as SDL

2/ Test : alpha-blended sprites
SDL  displayed 24 frames
SFML displayed 69 frames
--> SFML is 2.9x as fast as SDL

3/ Test : rotating sprites
SDL  displayed 14 frames
SFML displayed 71 frames
--> SFML is 5.1x as fast as SDL

4/ Test : static text
SDL  displayed 804 frames
SFML displayed 2566 frames
--> SFML is 3.2x as fast as SDL

5/ Test : dynamic text
SDL  displayed 119 frames
SFML displayed 2300 frames
--> SFML is 19.3x as fast as SDL

Test from http://www.sfml-dev.org/temp/bench-sdl-sfml.zip

[insa]

Testing Machine: lenovo S10e (Intel Atom N270 @ 1.60GHz, 2GB RAM), Win7

Code: [Select]

1/ Test : sprites
SDL  displayed 27 frames
SFML displayed 48 frames
--> SFML is 1.8x as fast as SDL

2/ Test : alpha-blended sprites
SDL  displayed 12 frames
SFML displayed 53 frames
--> SFML is 4.4x as fast as SDL

3/ Test : rotating sprites
SDL  displayed 6 frames
SFML displayed 51 frames
--> SFML is 8.5x as fast as SDL

4/ Test : static text
SDL  displayed 479 frames
SFML displayed 1110 frames
--> SFML is 2.3x as fast as SDL

5/ Test : dynamic text
SDL  displayed 33 frames
SFML displayed 464 frames
--> SFML is 14.1x as fast as SDL

[nullsquared]

I don't understand the point of this test.

It's like testing a software renderer vs. Direct3D and concluding that Direct3D is faster.

[K-Bal]

I don't understand the point of this test.

It's like testing a software renderer vs. Direct3D and concluding that Direct3D is faster.

An obvious conclusion, but some people don't know about this. So what's wrong with it?