Welcome, Guest. Please login or register. Did you miss your activation email?

Author Topic: Unit tests  (Read 17620 times)

0 Members and 1 Guest are viewing this topic.

Tank

  • Moderator
  • Hero Member
  • *****
  • Posts: 1483
    • View Profile
    • Blog
    • Email
Unit tests
« on: August 26, 2014, 10:01:34 am »
Hey guys,

I finally had some time to add the Catch framework together with one unit test example to the repository. You can find it here: https://github.com/SFML/SFML/commits/feature/unittest

I've used the BDD style as I find it to be incredibly readable. I think we should agree on the style first before I continue writing tests. I don't want to change millions of test cases. ;)

Laurent

  • Administrator
  • Hero Member
  • *****
  • Posts: 32504
    • View Profile
    • SFML's website
    • Email
Re: Unit tests
« Reply #1 on: August 26, 2014, 10:10:10 am »
It looks really good :)

I don't know the Catch framework so I trust you for the style. However I'm curious to see how it looks with more complex test cases.

Also, have you started to think about the continuous integration server, and about a framework for testing graphical stuff? (what? I'm asking too much? ;D)
Laurent Gomila - SFML developer

Tank

  • Moderator
  • Hero Member
  • *****
  • Posts: 1483
    • View Profile
    • Blog
    • Email
Re: Unit tests
« Reply #2 on: August 26, 2014, 10:53:10 am »
Quote from: Laurent
I don't know the Catch framework so I trust you for the style.
Catch supports another style as well, which is more technical. BDD has the advantage of being very readable, expressive and descriptive, and it makes you think about your problem ("What do I really need?").

Quote from: Laurent
However I'm curious to see how it looks with more complex test cases.
You are not the only one who's curious. ;) One issue is that SFML hasn't been developed with testability in mind, so it will be quite difficult in some spots. Maybe some refactorings here and there will be required. But we'll see how that will come along.

Quote from: Laurent
Also, have you started to think about the continuous integration server, and about a framework for testing graphical stuff?
You mean this? ;) It's right now building the master branch, but I'd like to change the infrastructure a bit to make it better extendable. zsbzsb will be contributing a FreeBSD box. A Windows 7 box is already available, but not yet connected to CI.

Quote from: Laurent
(what? I'm asking too much? ;D)
No. :)

Laurent

  • Administrator
  • Hero Member
  • *****
  • Posts: 32504
    • View Profile
    • SFML's website
    • Email
Re: Unit tests
« Reply #3 on: August 26, 2014, 10:58:35 am »
Quote
You mean this?
Ooohhh... it seems that everybody but me knew it. Was it announced somewhere? :P

I still have a Windows 7 and a Mac OS X 10.7 available if you need.
Laurent Gomila - SFML developer

Hiura

  • Moderator
  • Hero Member
  • *****
  • Posts: 4321
    • View Profile
    • Email
Re: Unit tests
« Reply #4 on: August 26, 2014, 10:59:23 am »
Good job!

There's something wrong on OS X though. But that's my headache, not yours.  ;)
(click to show/hide)

I've just one question: how safe is it to modify a variable that is used in several WHEN cases? I'm more specifically thinking of line 144 and 166 and how they might mess with each other. Does the Catch framework take care of this with some black magic?

You mean this? ;)
Wow! That's nice :D
SFML / OS X developer

Laurent

  • Administrator
  • Hero Member
  • *****
  • Posts: 32504
    • View Profile
    • SFML's website
    • Email
Re: Unit tests
« Reply #5 on: August 26, 2014, 11:05:25 am »
Quote
I've just one question: how safe is it to modify a variable that is used in several WHEN cases? I'm more specifically thinking of line 144 and 166 and how they might mess with each other. Does the Catch framework take care of this with some black magic?
I was reading the tutorial so I can answer your question: each WHEN triggers a new call (don't know from where it restarts), so they are self-contained and cannot mess up other sections.
Laurent Gomila - SFML developer

Tank

  • Moderator
  • Hero Member
  • *****
  • Posts: 1483
    • View Profile
    • Blog
    • Email
Re: Unit tests
« Reply #6 on: August 26, 2014, 11:09:37 am »
@Hiura
That error is probably related to the linker not being able to find the library files. Try adding a link_directories( "${PROJECT_BIN_PATH}/lib" ). Just a quick shot, though. ;)

As Laurent said, it's safe to use the variables like that -- it's actually encouraged and a replacement for the otherwise traditional setup/tear down machanism (which is very annoying to write, to be honest).

Hiura

  • Moderator
  • Hero Member
  • *****
  • Posts: 4321
    • View Profile
    • Email
Re: Unit tests
« Reply #7 on: August 26, 2014, 11:12:09 am »
Okay! That's a nice feature   :)

@Tank, unfortunately it doesn't solve the problem but I might know why. I'll have a look probably next week.
SFML / OS X developer

Hiura

  • Moderator
  • Hero Member
  • *****
  • Posts: 4321
    • View Profile
    • Email
Re: Unit tests
« Reply #8 on: October 03, 2014, 08:24:01 am »
...And I completely forgot about that... -> on my TODO list!

BTW, do we have a OS X system setup for the automatic build? If needed I could run from time to time a VM on my laptop that would communicate with the server (if someone explain me a bit how to do it of course). I guess I could launch it more than once a week (except for holidays). Would that be a good thing or do we look for a 100% uptime build machine?
SFML / OS X developer

Tank

  • Moderator
  • Hero Member
  • *****
  • Posts: 1483
    • View Profile
    • Blog
    • Email
Re: Unit tests
« Reply #9 on: October 03, 2014, 08:40:17 am »
Every interval is better than no interval, but having CI would be the best, of course. ;)

Hiura

  • Moderator
  • Hero Member
  • *****
  • Posts: 4321
    • View Profile
    • Email
Re: Unit tests
« Reply #10 on: October 05, 2014, 07:26:12 pm »
Alright, I've managed to make them run on OS X. You can find the changed on the feature/unittest_osx branch. There's two commits: the first one fix the runtime settings on OS X (specifically, the search path for binaries is more flexible) and the second one fix the Xcode templates. I've also rebase the branch against master.

EDIT: don't rush in merging anything into feature/unittest. I just realised that I might need to integrated these commits to the no_libsndfile branch too if I don't want to do it twice. In this case the better would be to add them directly into master and rebase both branches. I'll come back to that in due time.

I expect the installation procedure of JSFML and SFML.net to change a bit but for a regular usage of SFML there's nothing extra to do.

Regarding `SFML_BUILD_TEST`, I would recommend setting the default value to false. Regular user should not be that interested in tests I think.

And finally, about this CI thing, I'm happy to share some of my CPU time. I just need to know how to do it. Or at least, where to start looking for information.  :)
« Last Edit: October 05, 2014, 07:47:02 pm by Hiura »
SFML / OS X developer

Tank

  • Moderator
  • Hero Member
  • *****
  • Posts: 1483
    • View Profile
    • Blog
    • Email
Re: Unit tests
« Reply #11 on: October 06, 2014, 01:55:42 am »
Thanks for the commits. As you suggested I'll wait for them to enter master. If that's not going to happen, just drop me a line and I'll integrate them. :)

Quote
Regarding `SFML_BUILD_TEST`, I would recommend setting the default value to false. Regular user should not be that interested in tests I think.
I thought about that but chose to do it like I've been doing it in all projects: Default = on. The reason is that what's tests are for. If there's something failing, then chances are relatively high that we will receive a report on the forums or bug tracker instead of an unrecognized bug.

The compile times are higher (especially when the test suite will be more complete), but there are no extra steps involved for the user, so I'd still suggest to run them by default.

Hiura

  • Moderator
  • Hero Member
  • *****
  • Posts: 4321
    • View Profile
    • Email
Re: Unit tests
« Reply #12 on: October 07, 2014, 11:55:31 am »
FYI I've removed feature/unittest_osx to prevent any confusion.

About SFML_BUILD_TEST, what do the others think? I'm curious.  :)
SFML / OS X developer

Laurent

  • Administrator
  • Hero Member
  • *****
  • Posts: 32504
    • View Profile
    • SFML's website
    • Email
Re: Unit tests
« Reply #13 on: October 07, 2014, 12:04:12 pm »
Tank, do you mean that unit tests will be run automatically as part of the compilation process of SFML? If so, people will hardly notice them, and we'll get very precise and complete reports in case something is wrong. However if they must be run manually, then I'm against compiling/installing them by default.
Laurent Gomila - SFML developer

Tank

  • Moderator
  • Hero Member
  • *****
  • Posts: 1483
    • View Profile
    • Blog
    • Email
Re: Unit tests
« Reply #14 on: October 07, 2014, 12:28:03 pm »
They will indeed be part of the build process, no manual steps required.