Welcome, Guest. Please login or register. Did you miss your activation email?

Author Topic: Unit tests  (Read 34141 times)

0 Members and 1 Guest are viewing this topic.

Nexus

  • Moderator
  • Hero Member
  • *****
  • Posts: 6286
  • Thor Developer
    • View Profile
    • Bromeon
Re: Unit tests
« Reply #15 on: October 07, 2014, 03:00:32 pm »
You mean running the tests as part of the build, not only compiling them, right?

If so, this requires a script invoked as a post-build step, we probably would have to investigate how this can be done with CMake. But I'm not sure if it's very wise to run tests automatically, because people might not expect new applications to start when just compiling, and depending on the test, interaction with external file resources or even sockets would be necessary.
Zloxx II: action platformer
Thor Library: particle systems, animations, dot products, ...
SFML Game Development:

eXpl0it3r

  • SFML Team
  • Hero Member
  • *****
  • Posts: 10800
    • View Profile
    • development blog
    • Email
Re: Unit tests
« Reply #16 on: October 07, 2014, 03:03:34 pm »
It's not hard to add that to CMake. Also it's not hard to add another switch, which is checked by default and can be unchecked by the user.
Running tests after compilation is by now more or less a standard behavior, if users don't expect that even after they didn't uncheck the test build marker, then it's nothing we should be concerned about.
Official FAQ: https://www.sfml-dev.org/faq.php
Official Discord Server: https://discord.gg/nr4X7Fh
——————————————————————
Dev Blog: https://duerrenberger.dev/blog/

Hiura

  • SFML Team
  • Hero Member
  • *****
  • Posts: 4321
    • View Profile
    • Email
Re: Unit tests
« Reply #17 on: October 07, 2014, 03:13:28 pm »
I feel there's a misunderstanding on what is already implemented. Currently, when you do cmake followed by `make` (and not `make test`) you get:
$ make
[  1%] Built target SFML
[ 13%] Built target sfml-system
[ 41%] Built target sfml-window
[ 50%] Built target sfml-network
[ 74%] Built target sfml-graphics
[ 84%] Built target sfml-audio
[ 85%] Built target ftp
[ 86%] Built target opengl
[ 86%] Built target pong
[ 87%] Built target shader
[ 89%] Built target sockets
[ 90%] Built target sound
[ 91%] Built target sound-capture
[ 94%] Built target voip
[ 95%] Built target window
[ 98%] Built target cocoa
[100%] Built target sfmltest
Running test suite...===============================================================================
All tests passed (30 assertions in 2 test cases)

[100%] Built target runtests
 

unless you set the SFML_TEST (or whatever its name is, I forgot) to FALSE.
SFML / OS X developer

Tank

  • Moderator
  • Hero Member
  • *****
  • Posts: 1486
    • View Profile
    • Blog
    • Email
Re: Unit tests
« Reply #18 on: October 07, 2014, 06:33:02 pm »
CMake provides us a platform-independent way of launching executables. And since we control the build, there's absolutely no issue with that.

Tests that are not being run are useless, that's why I enable them by default. There are no disadvantages for the user. He will probably even be happy to see it. ;)

Tank

  • Moderator
  • Hero Member
  • *****
  • Posts: 1486
    • View Profile
    • Blog
    • Email
Re: Unit tests
« Reply #19 on: October 07, 2014, 06:34:26 pm »
Sorry, to clarify:

Build by default? Yes
Run after build by default? Yes
Mark build as failed when tests fail? Yes

Laurent

  • Administrator
  • Hero Member
  • *****
  • Posts: 32504
    • View Profile
    • SFML's website
    • Email
Re: Unit tests
« Reply #20 on: October 07, 2014, 09:56:21 pm »
Quote
Build by default? Yes
Run after build by default? Yes
Mark build as failed when tests fail? Yes
Agreed at 100%.
Laurent Gomila - SFML developer

Laurent

  • Administrator
  • Hero Member
  • *****
  • Posts: 32504
    • View Profile
    • SFML's website
    • Email
Re: Unit tests
« Reply #21 on: October 12, 2014, 04:11:44 pm »
Any news about the Windows machines for the CI server?

It would be really nice to avoid this ;)
Laurent Gomila - SFML developer

Tank

  • Moderator
  • Hero Member
  • *****
  • Posts: 1486
    • View Profile
    • Blog
    • Email
Re: Unit tests
« Reply #22 on: October 12, 2014, 06:56:54 pm »
It's setup, but not yet configured for CI. Will do it in the upcoming week.

Nexus

  • Moderator
  • Hero Member
  • *****
  • Posts: 6286
  • Thor Developer
    • View Profile
    • Bromeon
Re: Unit tests
« Reply #23 on: April 14, 2015, 08:42:34 pm »
Recently, I have experimented with Catch, and it's really great. It offers a lot of flexibility and is very smart at interpreting the code. The only unfortunate thing is that its header-only nature is really nice to get things up and running quickly, but leads to unnecessarily long compile times in long-term projects.

Anyway, I would be willing to contribute with some test cases.

I personally think the BDD-style testing is too verbose if applied rigorously.
SCENARIO("Constructing sf::Vector2")
{
    WHEN("default constructor is used")
    {
        sf::Vector2i vector;

        THEN("x and y are 0")
        {
            CHECK(vector.x == 0);
            CHECK(vector.y == 0);
        }
    }
}
vs.
SECTION("Default-constructing sf::Vector2 yields (0, 0)")
{
    sf::Vector2i vector;
    CHECK(vector.x == 0);
    CHECK(vector.y == 0);
}

It is absolutely obvious that we expect members to be zero after default construction, and one annotation in the SECTION macro is more than enough to describe it. In many cases, the CHECK expressions alone would suffice. Writing it three times in text over 13 lines feels to me like
++i; // increment i
that is, unnecessary noise, which makes code harder to read and more difficult to maintain. The simple sf::Vector2i test takes already 220 lines. Imagine how this explodes for other classes that have 15 no-brainer setters/getters, when you have to repeat the same verbosity again and again.

What I really love about Catch is that code written with it can be tremendously concise -- unlike awful test frameworks where you have to write class hierarchies and spend an insane amount of time for nothing. I suggest we exploit this advantage where possible, and I'm sure I'm much more motivated to write test cases if I'm not dealing with so much boilerplate. I would end up copy-pasting anyway, and that's never a good sign ;)

There may be more complex cases where we really need to state what was meant. BDD style is one option, but keep in mind the WHEN/THEN macros are semantically equivalent to SECTIONs, so we can achieve the same either way.

Edit: Even more extreme example:
        WHEN("equal vectors are tested for equality")
        {
            bool equal = (firstEqualVector == secondEqualVector);

            THEN("the result is true")
            {
                CHECK(equal == true);
            }
        }
vs.
SECTION("operator==") // do we really need to explain == semantics?
{
    CHECK(firstEqualVector == secondEqualVector);
}

Both the SECTION caption and the expression given to CHECK are visible in the output if the test fails, so that's already 2 times obvious :D

And apart from that, the second code is better because it contains the objects to check in the CHECK macro, allowing Catch to evaluate the operand's values. One would need to define a string output for sf::Vector2, but then one would immediately see where the implementation is broken aka "(3,3) == (3,3)", rather than just an unhelpful "false == true".
« Last Edit: April 14, 2015, 08:58:52 pm by Nexus »
Zloxx II: action platformer
Thor Library: particle systems, animations, dot products, ...
SFML Game Development:

binary1248

  • SFML Team
  • Hero Member
  • *****
  • Posts: 1405
  • I am awesome.
    • View Profile
    • The server that really shouldn't be running
Re: Unit tests
« Reply #24 on: April 14, 2015, 08:53:43 pm »
Nothing prevents us from writing our own wrapper/helpers/macros around Catch that makes our lives easier... If/When I eventually also start writing tests (e.g. for GL stuff), I would also prefer the second variant over the first one. Copy and pasting might make life easier, but if it has to always be done anyway, why not just alleviate the need for it all together? ;)
SFGUI # SFNUL # GLS # Wyrm <- Why do I waste my time on such a useless project? Because I am awesome (first meaning).

Tank

  • Moderator
  • Hero Member
  • *****
  • Posts: 1486
    • View Profile
    • Blog
    • Email
Re: Unit tests
« Reply #25 on: April 15, 2015, 11:52:56 am »
Quote from: Nexus
The only unfortunate thing is that its header-only nature is really nice to get things up and running quickly, but leads to unnecessarily long compile times in long-term projects.
Have you split your runner file from your test cases? Because usually compiling the runner file (the one that contains the CATCH_CONFIG_MAIN macro) takes some time, only.

Quote from: Nexus
I personally think the BDD-style testing is too verbose if applied rigorously.
The verbosity is actually a strong point of BDD, but I agree that in many cases, the bloat is quite annoying. BDD helped me to learn a certain way of thinking of test cases. But now that I learned that, I don't need it anymore in favor of more compact test cases. ;-)

Just take care when naming your section names: The names should be unique.

Nexus

  • Moderator
  • Hero Member
  • *****
  • Posts: 6286
  • Thor Developer
    • View Profile
    • Bromeon
Re: Unit tests
« Reply #26 on: April 15, 2015, 01:40:42 pm »
Have you split your runner file from your test cases? Because usually compiling the runner file (the one that contains the CATCH_CONFIG_MAIN macro) takes some time, only.
Yes, I had it separately. I was able to mitigate the long compile times by using MSVC precompiled headers, but it's still a workaround. Unfortunately, I can't precompile <Catch.hpp>, because it comes in different versions for runner and test-case files, so I just precompiled the standard headers used everywhere.


The verbosity is actually a strong point of BDD, but I agree that in many cases, the bloat is quite annoying. BDD helped me to learn a certain way of thinking of test cases. But now that I learned that, I don't need it anymore in favor of more compact test cases. ;-)
Glad you see it like this :)

What do others think? Do you like the compact way of writing test sections like
SECTION("Default-constructing sf::Vector2 yields (0, 0)")
{
    sf::Vector2i vector;
    CHECK(vector.x == 0);
    CHECK(vector.y == 0);
}

Would be nice to find an agreement soon, then I can start writing some tests :)


Just take care when naming your section names: The names should be unique.
Are you sure? The tutorial says test names must be unique, it doesn't mention sections. Keeping section names locally unique is definitely a good idea though (locally = within the same test case).
Zloxx II: action platformer
Thor Library: particle systems, animations, dot products, ...
SFML Game Development:

Hiura

  • SFML Team
  • Hero Member
  • *****
  • Posts: 4321
    • View Profile
    • Email
Re: Unit tests
« Reply #27 on: April 15, 2015, 01:54:29 pm »
I'm on my phone right now so I'll keep it short but if needed I can extend on later.

I also quite like Catch. First, it's the only framework with which I hadn't any trouble (cppunit reporting false memory leak, gtest being just ugly, ...).

Regarding the style of tests, I also felt that BDD is too verbose. Regardless of the style the (optional) tagging system is also a good strength of catch.
SFML / OS X developer

Tank

  • Moderator
  • Hero Member
  • *****
  • Posts: 1486
    • View Profile
    • Blog
    • Email
Re: Unit tests
« Reply #28 on: April 15, 2015, 02:51:26 pm »
@Nexus
I suggest removing the expected test results from the section description. The test itself is readable enough, and in case of errors, the expected values will be printed anyway. So maybe something like "Default-constructing sf::Vector2".

I really wonder that it's so slow on your hand. I haven't had any difficulties with compile times yet (both GCC and Clang).

Nexus

  • Moderator
  • Hero Member
  • *****
  • Posts: 6286
  • Thor Developer
    • View Profile
    • Bromeon
Re: Unit tests
« Reply #29 on: April 26, 2015, 04:13:05 pm »
I rebased feature/unittest onto latest master and rewrote the Vector2 tests in a more concise way. Are you okay with this style?

Since there seems to be public interest in contributing, we should quickly agree on some basic rules for style and consistency. Please comment on this thread even if you're okay with everything.


Some proposals, please tell me where you disagree or what's missing:

1. A single Catch test case (TEST_CASE macro) should be used for a unit (class, group of functions, etc.) to test. I adapted the Vector2 example, which used 2 test cases.

2. Functionality is grouped hierarchically in Catch sections (SECTION macro). For example, the Vector2 example has the following hierarchy:
(click to show/hide)

3. Test case and section names are concise, but contain enough information to understand what is tested in their context. If C++ syntax (like operators) makes things clearer than text, use them. Avoid repetition of parent test cases/sections (e.g. don't mention "sf::Vector2" again inside the Vector2 test case).

4. For assertions, the CHECK macro is used. REQUIRE, which stops at the first failed assertion, is avoided. To check negative conditions, CHECK_FALSE is used. More info...


Where I'm not sure yet:

5. How do we test with external resources? Many SFML classes require files (textures, sounds, shaders...) to be loaded in order to function correctly. Is this a problem if the tests are run as part of the build command? Are there limitations to keep in mind?

6. Shall we write string conversions to debug failed assertions?
Zloxx II: action platformer
Thor Library: particle systems, animations, dot products, ...
SFML Game Development:

 

anything