Welcome, Guest. Please login or register. Did you miss your activation email?

Author Topic: Unit tests  (Read 34153 times)

0 Members and 1 Guest are viewing this topic.

eXpl0it3r

  • SFML Team
  • Hero Member
  • *****
  • Posts: 10801
    • View Profile
    • development blog
    • Email
Re: Unit tests
« Reply #30 on: April 26, 2015, 05:25:22 pm »
Looks fine to me. About the test, I wonder though what's with sf::Vector2u and sf::Vector2f?
Official FAQ: https://www.sfml-dev.org/faq.php
Official Discord Server: https://discord.gg/nr4X7Fh
——————————————————————
Dev Blog: https://duerrenberger.dev/blog/

Hiura

  • SFML Team
  • Hero Member
  • *****
  • Posts: 4321
    • View Profile
    • Email
Re: Unit tests
« Reply #31 on: April 26, 2015, 06:08:03 pm »
Looks nice.  ;)

Regarding 1), I would add a tag to the test case (it can be as simple as the module being tested). This is nice if when working on something you want to run only a subset of tests.

Example:
TEST_CASE("sf::Vector2 class", "[system]")

Regarding 5), I guess we could duplicate some resources inside test/.

As for 6), I'd say we don't do it unless there's a bug and it's actually useful to have the output, unless someone really want to spent his time doing that  :P. And if needed, I'd go with a Catch::toString overload instead of overloading operator <<.
SFML / OS X developer

Nexus

  • Moderator
  • Hero Member
  • *****
  • Posts: 6286
  • Thor Developer
    • View Profile
    • Bromeon
Re: Unit tests
« Reply #32 on: April 27, 2015, 11:11:11 am »
About the test, I wonder though what's with sf::Vector2u and sf::Vector2f?
Tank wrote the test for Vector2i and it should generalize to Vector2<T>, because there are no template specializations, so the type should behave the same.

There are of course cases where int and float behave differently, for example with division, floating precision (bigger floats are less precise in their decimal places), rounding errors and special values (NaN, infinity, ...). But I think for the idea of testing the Vector2<T> functionality it's okay.

Test coverage is not 100% anyway, since we cover only one particular value per operation. Consider
    SECTION("Comparison operations (two equal and one different vector)")
    {
        sf::Vector2i firstEqualVector(1, 5);
        sf::Vector2i secondEqualVector(1, 5);
        sf::Vector2i differentVector(6, 9);

        SECTION("vector == vector")
        {
            CHECK(firstEqualVector == secondEqualVector);
            CHECK_FALSE(firstEqualVector == differentVector);
        }

        SECTION("vector != vector")
        {
            CHECK(firstEqualVector != differentVector);
            CHECK_FALSE(firstEqualVector != secondEqualVector);
        }
    }

The following wrong implementation of operator!= would still pass this test:
template <typename T>
bool operator!= (const sf::Vector2<T>& lhs, const sf::Vector2<T>& rhs)
{
    return lhs.x != rhs.x;
}

Now, in my opinion we can be a bit pragmatic, especially for class templates like sf::Vector2 that have existed for ages, been used by thousands of people, and are very unlikely to change in the future.


Regarding 1), I would add a tag to the test case (it can be as simple as the module being tested). This is nice if when working on something you want to run only a subset of tests.
Good idea.


Regarding 5), I guess we could duplicate some resources inside test/.
Should we really waste the space (e.g. for music or fonts, it can quickly be a few MB)? Is Git smart regarding exact file duplicates?


As for 6), I'd say we don't do it unless there's a bug and it's actually useful to have the output, unless someone really want to spent his time doing that  :P.
Maybe one for the most fundamental types like vectors or rects are useful, but I don't think it makes sense to provide it for more complex classes like sf::Sprite, either. After all, string conversions are only invoked in a relational comparison, an operation that most SFML classes don't even support.
« Last Edit: April 27, 2015, 11:12:50 am by Nexus »
Zloxx II: action platformer
Thor Library: particle systems, animations, dot products, ...
SFML Game Development:

Hiura

  • SFML Team
  • Hero Member
  • *****
  • Posts: 4321
    • View Profile
    • Email
Re: Unit tests
« Reply #33 on: April 27, 2015, 01:40:11 pm »
Should we really waste the space (e.g. for music or fonts, it can quickly be a few MB)? Is Git smart regarding exact file duplicates?

No, git doesn't support such feature. But that shouldn't really be an issue: currently the whole examples directory, including source code and probably duplicate resources, weight ~1.5MB.

If someone really want to only download the head, without history, he can do a shallow clone with `--depth 1` which is result (today) in a <20 MB download instead of ~70MB (but he won't be able to push/pull in the future so it's simpler to download the zip archive directly).

 
SFML / OS X developer

Tank

  • Moderator
  • Hero Member
  • *****
  • Posts: 1486
    • View Profile
    • Blog
    • Email
Re: Unit tests
« Reply #34 on: April 28, 2015, 04:13:40 pm »
Quote
4. For assertions, the CHECK macro is used. REQUIRE, which stops at the first failed assertion, is avoided. To check negative conditions, CHECK_FALSE is used. More info...
REQUIRE is especially useful when you want to make sure that certain critical test conditions are met. E.g.:

foo.add(123);
foo.add(456);
REQUIRE(foo.getCount() == 2);
CHECK(foo[0] == 123);
CHECK(foo[1] == 456);


Quote
5. How do we test with external resources? Many SFML classes require files (textures, sounds, shaders...) to be loaded in order to function correctly. Is this a problem if the tests are run as part of the build command? Are there limitations to keep in mind?
Everything that's needed for testing must be provided. Keep in mind that test resources are usually really small. E.g. a sound file that's exactly 1 second long and contains a specific set of data.

Quote
6. Shall we write string conversions to debug failed assertions?
Why not? It may greatly improve the readability of failed test reports.

Nexus

  • Moderator
  • Hero Member
  • *****
  • Posts: 6286
  • Thor Developer
    • View Profile
    • Bromeon
Re: Unit tests
« Reply #35 on: April 28, 2015, 08:54:24 pm »
REQUIRE is especially useful when you want to make sure that certain critical test conditions are met.
That's a really good point. If I remember correctly, REQUIRE aborts the whole test case, not just the current section, right? It's not tragic, but we might miss some information about other failed tests. That is, when a user reports a failed test to us, remaining results might be handy for debugging.

E.g. a sound file that's exactly 1 second long and contains a specific set of data.
Maybe a more general question: how do we operationally test such multimedia features? We can of course access the raw data (audio samples, pixels, ...) to some extent.

But, do we check whether a resource has been loaded correctly, for example? Or test rendering... are there some state-of-the art techniques, or should we just sample some pixels and compare with hardcoded values? ;)

Anyway, I think there are enough other things to test before we get to this point, so it's not a blocking issue 8)
Zloxx II: action platformer
Thor Library: particle systems, animations, dot products, ...
SFML Game Development:

Hiura

  • SFML Team
  • Hero Member
  • *****
  • Posts: 4321
    • View Profile
    • Email
Re: Unit tests
« Reply #36 on: April 28, 2015, 09:28:42 pm »
No answer here, just more questions: How de we make sure the window is properly positioned, that an event is generated when the user clicks on his mouse, that the title is correct, ...

We probably don't have to go as far as testing the title but making sure the same events are generated oj all OSes could be good. Bu how?...
SFML / OS X developer

Nexus

  • Moderator
  • Hero Member
  • *****
  • Posts: 6286
  • Thor Developer
    • View Profile
    • Bromeon
Re: Unit tests
« Reply #37 on: April 28, 2015, 10:42:22 pm »
Hiura, if there's something we cannot meaningfully hardcode in the C++ source (like a collection of events or so), one possibility is also to store them in a text file, and let the test read this file on different OSes. But I agree that such things will be very difficult to test, especially since you can't easily simulate user input.

I added string conversions for the following classes. I tried to make the format rather concise while still being expressive enough.
  • String
  • Time
  • Vector2<T>
  • Vector3<T>
  • VideoMode
  • Color
  • Rect<T>
Keep in mind that it only makes sense to provide string conversions for types that support relational operators, or at least == and !=, as they're used in Catch assertion expressions. So don't start to write conversions for sf::Sprite or so.

Types that are still missing are all the enums, like mouse buttons, joystick axes, keyboard keys, event types, and so on. But enums are a pain to serialize in C++, as you can essentially perform a case differentiation for every single enumerator. I did this in Thor for some of them... But I think here I'll wait until we actually need them, otherwise we just have to keep it in sync with the original enums (<- this maintenance is the problem).

Furthermore, I introduced a new header UnitTests.hpp which is used by all the test cases. It indirectly includes <catch.hpp>. Don't include <catch.hpp> directly, or else you will not benefit from string conversions.

I also added tags in TEST_CASE.

Any opinions on these changes? What do you think about the enums?
« Last Edit: April 28, 2015, 10:43:57 pm by Nexus »
Zloxx II: action platformer
Thor Library: particle systems, animations, dot products, ...
SFML Game Development:

Tank

  • Moderator
  • Hero Member
  • *****
  • Posts: 1486
    • View Profile
    • Blog
    • Email
Re: Unit tests
« Reply #38 on: April 29, 2015, 08:28:36 am »
Good stuff. :) Regarding the enums: We can do it for things like EventType, for example, where it might be useful. But for enums like sf::Keyboard::Key I wouldn't really do it.

Tank

  • Moderator
  • Hero Member
  • *****
  • Posts: 1486
    • View Profile
    • Blog
    • Email
Re: Unit tests
« Reply #39 on: April 29, 2015, 08:34:28 am »
And regarding to what Hiura said:
This is currently not really testable, because SFML hasn't been developed with being easily testable in mind (in contrast to doing test-driven development, for example). What you usually do is using mock objects.

Instead of querying the actual OS for events, for example, or the mouse position, you use a mock object that has the same interface, but is controllable by the unit test. That way you can manually set a mouse position that the unit to be tested retrieves, and thus you can test if it functions properly. The same applies to nearly anything that requires external dependencies.

Hiura

  • SFML Team
  • Hero Member
  • *****
  • Posts: 4321
    • View Profile
    • Email
Re: Unit tests
« Reply #40 on: April 29, 2015, 08:50:40 am »
That 'mock' strategy should probably be taken into account for SFML 3, don't you think?
SFML / OS X developer

Tank

  • Moderator
  • Hero Member
  • *****
  • Posts: 1486
    • View Profile
    • Blog
    • Email
Re: Unit tests
« Reply #41 on: April 29, 2015, 09:37:24 am »
Yeah.

Nexus

  • Moderator
  • Hero Member
  • *****
  • Posts: 6286
  • Thor Developer
    • View Profile
    • Bromeon
Re: Unit tests
« Reply #42 on: April 29, 2015, 12:56:35 pm »
Wouldn't that require a massive internal rewrite? An additional intermediate abstraction layer just for mocking would increase the complexity and with it the maintenance effort for all times. And don't forget that some operations have to be fast, we can't just add a virtual function call in every SFML operation ;)

Considering that most event- and input-related bugs appear because of different behavior of all the platform-specific APIs and not because of mistakes in SFML's front-end, I'm not even sure how much mocking would help.
Zloxx II: action platformer
Thor Library: particle systems, animations, dot products, ...
SFML Game Development:

Hiura

  • SFML Team
  • Hero Member
  • *****
  • Posts: 4321
    • View Profile
    • Email
Re: Unit tests
« Reply #43 on: April 29, 2015, 01:01:48 pm »
By "taken into account" I didn't mean "implemented for everything".  ;)

Sure, some things might not be well suited for such testing, but some do. Now, I'm not experienced enough with TDD to tell off the top of my head which part could benefit from that. Maybe some of you have more intuition about that?
SFML / OS X developer

Tank

  • Moderator
  • Hero Member
  • *****
  • Posts: 1486
    • View Profile
    • Blog
    • Email
Re: Unit tests
« Reply #44 on: April 29, 2015, 01:27:43 pm »
@Nexus:

Yeah, that requires a massive rewrite. Improving the whole codebase step-by-step is alright though. It's not that you have to do it all at once.

Automated tests for some areas are for sure very difficult to do, like rendered stuff etc. that behaves differently on different hardware.

Your concerns about slowing down certain areas are of course also valid. That's why you sometimes have to apply tricks and hacks to be able to exchange external dependencies. Unfortunately this can be very tough with both C++ and C, because things like overwriting symbols from outside is not possible. That pretty much leaves you with techniques like swapping external libraries with a mock replacement, or adding preprocessor directives to the internal code that allows to hook in mocking services.

100% test coverage should be the ultimate goal. We'll see what problems we'll encounter and what options exist to eliminate them. :)

@Hiura:

I didn't understand your question. :(

 

anything