Unit Testing

[Azaral]

I just recently came across this. I only have a vague idea of what it is. I have no idea how to actually do it, but I would like to give it a try, but I have no idea where to being or even what is really going on with one. Any kind of information you could give me would be great.

[eXpl0it3r]

Well in our age it should be clear what to do when you've no idea on a topic: google it! Start with Wikipedia then look up different unit testing tools (for C++), then write some tests.

You might also find TDD (Test Driven Development) interesting.

Personally, I've only ever worked with boosts unit testing.

*summons MorleyDev*

[Azaral]

I have been googling it. I was hoping to get some other info from other people. I tend to not be very good at finding good information with google lol.

[eXpl0it3r]

Well if you have a specific question ask away. For a general definition and overview, we can't really beat Wikipedia.

There are many ways to implement/use unit testing, the basic idea is that for every "unit" you write, you a test and at best you even write the test before writing the code. And a "unit" is just the smallest testable code part.

[wintertime]

Most you can read about unit-testing is dry stuff. But there is a simple example for the simplest test library I found (very easy to write tests, no manual registering of tests needed), which I am using: http://unittest-cpp.sourceforge.net/money_tutorial/.
Btw., they moved to github if you want to try the newest version: https://github.com/unittest-cpp/unittest-cpp.

[MorleyDev]

TDD is very nice. I've worked in the business industry at a firm that did extensive TDD from high level acceptance testing right down to the lowest of unit tests (creating a fractal of tests). It's incredibly effective when done right, when you allow the tests to shape your code instead of trying to hammer the tests to fit your designs.

I'm a pretty big advocate of the practice, and that it's not trivially mockable is one of my biggest gripes with SFML (but that's a whole other can of worms).

UnitTest++ is nice but showing it's age and era, as is GoogleTest and Boost.Test. Personally I'm an advocate of the more recently released libraries (and even wrote my own, http://github.com/MorleyDev/UnitTest11) since they take advantage of C++11 features like lambdas, and as such remove the macro-heavy uglyness of older testing libraries.

http://banditcpp.org/, for example, looks very nice and I'd recommend it over my own library. It's better documented and looks prettier, I've not had time to do much work on mine lately and need to get back to it at some point.

The one advantage I think my UnitTest11 has that I should probably push more is it comes with a pretty decent mocking library integrated into it. It doesn't automock since this cannot yet be done cross-platform. I would love for a new version of C++ to include compile-time reflection that would allow this functionality...

[Tank]

I'm also a big fan of test-driven development. Since I learned what it is and how it works, I haven't been programming in any other way than TDD.

I wrote two paragraphs about testing in general and TDD in one of my GDD articles, which you can find here: http://stefan.boxbox.org/2013/02/10/game-development-design-2-good-habits-techniques-and-workflows/

As for the framework, I personally have deeper experience with Boost's Unit Test Framework and CATCH, whereas I love the latter one! (As a bonus it's header-only!)

I've tested a lot of other frameworks, for example CppUnit and Google's Test, but I found almost all of them to be way too specific in their features. For example it's completely unnecessary to have different comparison functions for "greater than" and "equals", instead C++ operators should be used. Also I really don't like a lot of unit testing boilerplate code. In my opinion unit testing should concentrate on the test cases, and not on how to design them.

"Bandit" (mentioned by MorleyDev, thanks!) sounds interesting, but I personally haven't looked at it in detail yet.

One thing's for sure, though: Once you get into TDD, you will never leave it again -- at least without feeling bad.

[Azaral]

Thanks for sharing the Catch testing tank. I'm giving it a go in a little simulator I'm making for some physics stuff. So far I'm really liking it. I can know for certain that my functions are doing what I want them to do without having to run the program and see results.

[Tank]

Yes, and whatever you change, you can always make sure that nothing breaks, at anytime (this requires an ideal code coverage of 100%, of course; which you can reach by doing TDD, i.e. by only writing code for existing tests, not the other way around).

[Nexus]

Yes, and whatever you change, you can always make sure that nothing breaks, at anytime (this requires an ideal code coverage of 100%, of course; which you can reach by doing TDD, i.e. by only writing code for existing tests, not the other way around).

Isn't this a bit idealistic? You can never cover everything in tests, and waiting for all possible tests to be written before a feature is implemented is only feasible in simple cases. You can exactly write tests for the functionality you know you're currently testing, but bugs often appear in unexpected places (such as when you combine multiple features, and only in rare special cases). I just think given a certain complexity, it's unreasonable to assume complete bug-freeness, but of course you can reduce them to a minimum with techniques like TDD.

[eXpl0it3r]

Isn't this a bit idealistic? You can never cover everything in tests, and waiting for all possible tests to be written before a feature is implemented is only feasible in simple cases. You can exactly write tests for the functionality you know you're currently testing, but bugs often appear in unexpected places (such as when you combine multiple features, and only in rare special cases). I just think given a certain complexity, it's unreasonable to assume complete bug-freeness, but of course you can reduce them to a minimum with techniques like TDD.

I guess it's important to understand that 100% code coverage doesn't mean the application is bug free. Yes you can never account for all possible things that could happen, especially when it comes to user errors, but the idea behind TDD is, that you don't write code to test, but you write tests to code. That way every piece of code you end up with has a test to it (or rather the other way around), which essentially means 100% code coverage, but still doesn't guarantee that there are no bugs, the chances of unintentional bugs just get reduced quite a bit and since you have all these tests, figuring out what is not being tested is quite a bit easier.

Personally, I think it sounds awesome, but it often seems like so much work, plus there's always that thing, that you can't really test, especially for games...

[Tank]

Isn't this a bit idealistic?

No, absolutely not.

You can never cover everything in tests

You can, and by applying TDD, you choose a workflow that even forces that. There are features that can't be covered by automated unit tests though, for example graphical things. Those have to be tested by humans but  are also usually not critical.

and waiting for all possible tests to be written before a feature is implemented is only feasible in simple cases

I think you got the idea of TDD wrong. You don't write all tests beforehand. You write one single test case, and then you write the implementation, like this:

// Test case:
test( sum( 1, 2 ) == 3 );

// Implementation:
int sum( int a, int b ) {
  return a + b;
}

One rule is that you only write code in the implementation that's being covered by the test cases. If it's not covered, you have to write another test first.

but bugs often appear in unexpected places (such as when you combine multiple features, and only in rare special cases)

And that's what unit testing avoids: Having unexpected bugs. If code is covered, there's no way you can miss a bug. You can still write wrong test cases, but here the following applies: Do you want to not do testing and accept a high chance of bugs, or do you want to do testing and accept a very low chance of writing wrong tests? If something's goes wrong, you've got a problem in both cases, but with unit testing it's minimized.

When you combine multiple features you do so called "integration tests", either with real objects or mock objects. It's all testable.

the chances of unintentional bugs just get reduced quite a bit and since you have all these tests, figuring out what is not being tested is quite a bit easier.

What do you mean by "unintentional bugs"? Let's assume theoretical 100% code coverage, how should a bug happen? The only chance something goes wrong is when you write a wrong test case -- but that's not a bug, it's a logic error.

Personally, I think it sounds awesome, but it often seems like so much work, plus there's always that thing, that you can't really test, especially for games...

Writing unit tests means work, yes. Robert C. Martin says "Professional programmers want to make sure that their code works", and I fully agree to that. The result is software tests. And by the way, software tests are also fun once you get used to the workflow.

The situation for games is not really different than for other application types. You can test nearly everything, except graphics effects and such. Untestable things are the rare cases.

[wintertime]

I think you can still have bugs if you have covered 100% of the code with some test, but did not handle 100% of the functionality in the tests.

// Test case:
test( sum( 1, 2 ) == 3 );

// Implementation:
int sum( int a, int b ) {
  return 3;
}
 

Oh yeah, I read you should only add the simplest thing to satisfy the test.
Also you would be supposed to make the test fail before making it pass. And then add more tests to get to the real implementation.
Somehow its easiest for those simple functions that always get put into examples and much work for something that rarely could get wrong. But the things you would gain most from having tests for are also the most difficult to write the tests for (thats what I'm still struggling with).

[Kojay]

The benefits of TDD are well established. On the other hand, I 'm sceptical/unclear on how one is meant to aim for 100% coverage when developing a game, due to visual aspect. How do you verify that what you mean to draw really is drawn? As in gui design, running the application as a user is a far more straightforward way to test (by the way, correct me if I 'm wrong Tank but SFGUI doesn't got a test suite; instead the examples are used as tests).

Incidentally, there is a new version of Boost.Test that only requires docs to make it to release - though I have not tried it and don't believe it would make use of C++11.

[Tank]

int sum( int a, int b ) {
  return 3;
}

Oh yeah, I read you should only add the simplest thing to satisfy the test.

That's true, but why on earth do you want to trick yourself? ;-) That's what a lot of people get wrong as well: Unit testing doesn't mean to find ways to make everything fail, it's there to make sure that the stuff you write works as expected.

Returning a constant 3 in a function named "sum()" is clearly a logic error. If that's what you intended, then it's 100% bug-free. Again, illogical stuff is not a bug!

Also you would be supposed to make the test fail before making it pass.

Yes, that's true and important.

But the things you would gain most from having tests for are also the most difficult to write the tests for (thats what I'm still struggling with).

Can you give an example? I'm willing to help.

how one is meant to aim for 100% coverage when developing a game, due to visual aspect. How do you verify that what you mean to draw really is drawn?

The actual rendering itself is a very small part of the whole visual thing. For example you can test that positions, properties, blending modes and whatnot of your rendering objects are as you would expect. The final image has to be approved by human eyes; if you then continue by grabbing a screenshot of the result and use that for comparing against future final results is up to you, but for some things it's really okay to move them to the acceptance test area. ;-)

correct me if I 'm wrong Tank but SFGUI doesn't got a test suite; instead the examples are used as tests

It's correct. I've started SFGUI when I wasn't into software tests. And I can tell you that so many bugs could have been avoided by doing tests. Besides of that it's really time-consuming and annoying to run through all the examples (often the "test" program is enough, because it literally contains everything), click everything, check everything, look at everything etc. Side effects happen, and they are dangerous (they happen less with TDD, because you design differently; but if they happen, you usually detect them through failing tests somewhere).