Unit tests

[Tank]

Hey guys,

I finally had some time to add the Catch framework together with one unit test example to the repository. You can find it here: https://github.com/SFML/SFML/commits/feature/unittest

I've used the BDD style as I find it to be incredibly readable. I think we should agree on the style first before I continue writing tests. I don't want to change millions of test cases.

[Laurent]

It looks really good

I don't know the Catch framework so I trust you for the style. However I'm curious to see how it looks with more complex test cases.

Also, have you started to think about the continuous integration server, and about a framework for testing graphical stuff? (what? I'm asking too much? )

[Tank]

I don't know the Catch framework so I trust you for the style.

Catch supports another style as well, which is more technical. BDD has the advantage of being very readable, expressive and descriptive, and it makes you think about your problem ("What do I really need?").

However I'm curious to see how it looks with more complex test cases.

You are not the only one who's curious. One issue is that SFML hasn't been developed with testability in mind, so it will be quite difficult in some spots. Maybe some refactorings here and there will be required. But we'll see how that will come along.

Also, have you started to think about the continuous integration server, and about a framework for testing graphical stuff?

You mean this? It's right now building the master branch, but I'd like to change the infrastructure a bit to make it better extendable. zsbzsb will be contributing a FreeBSD box. A Windows 7 box is already available, but not yet connected to CI.

(what? I'm asking too much? )

No.

[Laurent]

You mean this?

Ooohhh... it seems that everybody but me knew it. Was it announced somewhere?

I still have a Windows 7 and a Mac OS X 10.7 available if you need.

[Hiura]

Good job!

There's something wrong on OS X though. But that's my headache, not yours. 

(click to show/hide)

Scanning dependencies of target sfmltest
[ 98%] [100%] Building CXX object test/CMakeFiles/sfmltest.dir/src/Main.cpp.o
Building CXX object test/CMakeFiles/sfmltest.dir/src/Vector2.cpp.o
Linking CXX executable sfmltest
[100%] Built target sfmltest
Scanning dependencies of target runtests
Running test suite...dyld: Library not loaded: @executable_path/../Frameworks/libsfml-graphics.2.dylib
  Referenced from: /Users/m/Prog/Libraries/C++/SFML/git/builds/unittest/test/sfmltest
  Reason: image not found
/bin/sh: line 1: 36389 Trace/BPT trap: 5       /Users/m/Prog/Libraries/C++/SFML/git/builds/unittest/test/sfmltest
make[2]: *** [test/CMakeFiles/runtests] Error 133
make[1]: *** [test/CMakeFiles/runtests.dir/all] Error 2
make: *** [all] Error 2
 

I've just one question: how safe is it to modify a variable that is used in several WHEN cases? I'm more specifically thinking of line 144 and 166 and how they might mess with each other. Does the Catch framework take care of this with some black magic?

You mean this?

Wow! That's nice

[Laurent]

I've just one question: how safe is it to modify a variable that is used in several WHEN cases? I'm more specifically thinking of line 144 and 166 and how they might mess with each other. Does the Catch framework take care of this with some black magic?

I was reading the tutorial so I can answer your question: each WHEN triggers a new call (don't know from where it restarts), so they are self-contained and cannot mess up other sections.

[Tank]

@Hiura
That error is probably related to the linker not being able to find the library files. Try adding a link_directories( "${PROJECT_BIN_PATH}/lib" ). Just a quick shot, though.

As Laurent said, it's safe to use the variables like that -- it's actually encouraged and a replacement for the otherwise traditional setup/tear down machanism (which is very annoying to write, to be honest).

[Hiura]

Okay! That's a nice feature   

@Tank, unfortunately it doesn't solve the problem but I might know why. I'll have a look probably next week.

[Hiura]

...And I completely forgot about that... -> on my TODO list!

BTW, do we have a OS X system setup for the automatic build? If needed I could run from time to time a VM on my laptop that would communicate with the server (if someone explain me a bit how to do it of course). I guess I could launch it more than once a week (except for holidays). Would that be a good thing or do we look for a 100% uptime build machine?

[Tank]

Every interval is better than no interval, but having CI would be the best, of course.

[Hiura]

Alright, I've managed to make them run on OS X. You can find the changed on the feature/unittest_osx branch. There's two commits: the first one fix the runtime settings on OS X (specifically, the search path for binaries is more flexible) and the second one fix the Xcode templates. I've also rebase the branch against master.

EDIT: don't rush in merging anything into feature/unittest. I just realised that I might need to integrated these commits to the no_libsndfile branch too if I don't want to do it twice. In this case the better would be to add them directly into master and rebase both branches. I'll come back to that in due time.

I expect the installation procedure of JSFML and SFML.net to change a bit but for a regular usage of SFML there's nothing extra to do.

Regarding `SFML_BUILD_TEST`, I would recommend setting the default value to false. Regular user should not be that interested in tests I think.

And finally, about this CI thing, I'm happy to share some of my CPU time. I just need to know how to do it. Or at least, where to start looking for information. 

[Tank]

Thanks for the commits. As you suggested I'll wait for them to enter master. If that's not going to happen, just drop me a line and I'll integrate them.

Regarding `SFML_BUILD_TEST`, I would recommend setting the default value to false. Regular user should not be that interested in tests I think.

I thought about that but chose to do it like I've been doing it in all projects: Default = on. The reason is that what's tests are for. If there's something failing, then chances are relatively high that we will receive a report on the forums or bug tracker instead of an unrecognized bug.

The compile times are higher (especially when the test suite will be more complete), but there are no extra steps involved for the user, so I'd still suggest to run them by default.

[Hiura]

FYI I've removed feature/unittest_osx to prevent any confusion.

About SFML_BUILD_TEST, what do the others think? I'm curious. 

[Laurent]

Tank, do you mean that unit tests will be run automatically as part of the compilation process of SFML? If so, people will hardly notice them, and we'll get very precise and complete reports in case something is wrong. However if they must be run manually, then I'm against compiling/installing them by default.

[Tank]

They will indeed be part of the build process, no manual steps required.