OUnit 2.0 progress, September 2013

Continuing last month progress report onOUnit2. The release is just a few days away, I am testing real life application and the core of the work is already in the VCS.

The basic features:

  • better configuration setup
    • environment variable
    • command line options
    • configuration files
  • improved output of the tests:
    • allow vim quickfix to jump in the log file where the error has happened
    • output HTML report
    • output JUnit report
    • systematic logging (verbose always on), but output log in a file
  • choose how to run your test:
    • run tests in parallel using processes (auto-detect number of CPU and run as many worker processes)
    • run tests concurrently using threads
    • use the old sequential runner
  • choose which test to run with a chooser that can do smart selection of tests:
    • simple: just run test in sequence
    • failfirst: run the tests that failed in the last run first and skip the success if they are still failing
  • some refactoring:
    • bracket: use a registration in the context but easier to use
    • remove all useless functions in the OUnit2 interface
  • non-fatal section: allow to fail inside non-fatal section without quitting immediately the whole test
  • allow to use OUnit1 test inside OUnit2 (smooth the transition)
  • timer that makes tests fail if they take too long, only when using the processes runner (I was not able to do it cleanly using threads and sequential)
  • allow to parametrize filenames so that you can use OUNIT_OUTPUT_FILE=ounit-$(suite_name),log and have $(suite_name) replace by the test suite name
  • create locks to avoid accessing the same resources within a single process or the whole application (typically to avoid doing a chdir while another thread is doing a chdir elsewhere)
  • create a in_testdata_dir function to locate test data, if any

Still remaining to do, but quite straightforward:

  • sys admin (website, release process)
  • update the whole documentation

Some things that I decided not to do for OUnit 2.0 release:

  • introduce a 'cached' state to avoid rerunning a test if you can programmaticaly determine that the result will be the same.

The main development is now done, but before releasing I decided to test it first on a real scale application. The first big migration to OUnit2, will be the OASIS test suite. This is a pretty big test suite (100+ tests) that takes a fair amount of time to run. I hope that during the next week I will be able to port the whole test suite and come back with some timing results.

You can follow my progress on porting OASIS to OUnit 2.0, in github.

Help wanted

If you have a long standing issue with OUnit, this is the time to submit a bug. OUnit BTS

If you want to try the dev version of OUnit:

 $> darcs get http://forge.ocamlcore.org/anonscm/darcs/ounit/ounit

Patches always welcome.

You can read the documentation of the devel version the website.