Test Support Tools

copied from TiddlyWiki

This page holds content copied from the old "main" TiddlyWiki
It needs to be reworked, formatted and generally brought into line

Lumiera relies on test-driven development

This page discusses the overall organisation of test code, and the tools used for running test cases

Tests are the only form of documentation, known to provides some resilience against becoming outdated. Tests help to focus on the usage, instead of engaging in spurious implementation details. Developers are highly encouraged to write the tests before the actual implementation, or at least alongside and interleaved with expanding the feature set of the actual code. There may be exceptions to this rule. Not every single bit needs to be covered by tests. Some features are highly cross-cutting and exceptionally difficult to cover with tests. And sometimes, just an abstract specification is a better choice.

As a rule of thumb, consider to write test code which is easy to read and understand, like a narration to show off the relevant properties of the test subject.

Test Structure

a test case is an executable, expected to run without failure, and optionally producing some verifiable output
simple test cases may be written as stand-alone application, while the more tightly integrated test cases can be written as classes within the Lumiera application framework.
test cases should use the CHECK macro of NoBug to verify test results, since the normal assertions may be de-configured for optimised release builds.
our test runner script test.sh provides mechanisms to check for expected output

Several levels of aggregation are available. At the lowest level, a test typically runs several functions within the same test fixture. This allows to create a “narrative” in the code: first do this, than do that, and now that, and now this should happen… Generally speaking, it is up to the individual test to take care or isolate himself from any dependencies. Test code and application code uses the same mechanisms for accessing other components within the application. Up to now (2014), there was no need for any kind of dependency injection, nor did we face any difficulties with tainted state.

Test classes are organised into a tree closely mirroring the main application source code tree. Large sections of this test tree are linked together into test libraries. Some of these are linked against a specific (sub)scope of the application, like e.g. only against the support library, the application framework or the vault. Since we use strict dependencies, this linking step will spot code not being placed at the correct scope within the whole system. As a final step, the build system creates a test runner application (target/test-suite), which links dynamically against all the test libraries and thus against all application dependencies. Individual test classes integrate into this framework by placing a simple declaration (actually using the LAUNCHER macro), which typically also defines some tags and classification alongside. This way, using command line parameters for invocation of the test runners, it is possible to run some category or especially tagged test classes, or to invoke just a single test class in isolation (using the ID, which is also the class name).

The next level of aggregation is provided by the top level test collection definitions located in the test/ subdirectory. For running these collections as automatic tests within the build process, we use Cehteh’s test.sh shell script.

Tools and conventions

Test code and application code has to be kept separate; the application may be built without any tests, since test code has a tendency to bloat the executables, especially in debug mode. As an exception, generic test support code may be included in the library, and it is common for core components to offer dedicated test support and diagnostic features as part of the main application.

Conventions for the Buildsystem

to help with automating the build and test execution, test code should adhere to the following conventions:

test class names should end with the suffix _test
their tree and namespace location should correspond to the structure of the main application
test classes should be placed into a sub namespace …::test
specific definitions for a single test case may rely on global variables, but these should live within an anonymous namespace
all test executables should be named according to the pattern test-*
all test source code is within the tests subtree
immediately within the tests/ directory are the test collection definitions *.tests, ordered by number prefixes
below are the directories with test cases, to be grouped into the aforementioned test-executables.
we distinguish two ways to write and link tests: standalone and suite
- subtrees of test classes (C++) are linked into one shared library per subtree. In the final linking step, these are linked together into a single testrunner, which is also linked against the application core. The resulting executable test-suite is able to invoke any of the test classes in isolation, or a group / category of tests.
- simple plain-C tests (names starting with test-*) are grouped into several directories thematically, and linked according to the application layer. Each of those simple tests needs to be self contained and provide a main method.

Internal testsuite runner

The class test::Suite (as used by tests/testrunner.cpp) helps building an executable which will run all registered test case objects, or some group of such test cases. Each test case implements a simple interface and thus provides a run (args) function, moreover, it registers itself immediately alongside with his definition; this works by the usual trick of defining a static class object and calling some registration function from the constructor of this static var. See the following

hello-world-test example

#include "lib/test/run.hpp"
#include <iostream>

using std::cout;
using std::endl;

namespace test   {

  class HelloWorld_test
    : public Test
    {
      virtual void
      run (Arg)
        {
          greeting();
        }

      void
      greeting()
        {
          cout << "goodbye cruel world..." <<endl;
        }
    };

  /** Register this test class to be invoked in some test groups (suites) */
  LAUNCHER (HelloWorld_test, "unit function common");
}

Notes:

type Arg is compatible to std::vector<string> &
this vector may be arg.size()==0, which means no commandline args available.
these args may contain further arguments passed from system commandline (or the testsuite definition).
the test can/should produce output that can be checked with test.sh
the macro LAUNCHER expands to
Launch<HelloWorld_test> run_HelloWorld_test("HelloWorld_test","unit function common");
note the second parameter to the macro (or the Laucher-ctor) is a space-delimited list of group names
thus any test can declare itself as belonging to some groups, and we can create a test::Suite for each group if we want.

invoking a testrunner executable

The class test::TestOption predefines a boost-commandlineparser to support the following options:

`./test-suite --group <groupID> [testID [arguments...]]`
`--help`	options summary
`-g <groupID>`	run all tests from this group as suite. If missing, ALL tests will be included
`[testID]`	(optional) one single testcase. If missing, all testcases of the group will be invoked
`--describe`	print all registered tests to stdout in a format suited for use with test.sh

Further commandline arguments are deliverd to a single testcase only if you specify a testID. Otherwise, all commandline arguments remaining after options parsing will be discarded and all tests of the suite ill be run with an commandline vector of size()==0

The Test Script `test.sh`

To drive the various tests, we use the script tests/test.sh.
All tests are run under valgrind control by default (if available), unless VALGRINDFLAGS=DISABLE is defined (Valgrind is sometimes prohibitively slow). The buildsystem will build and run the testcode when executing the target scons check. The target scons testcode will just build but not execute any tests.

Options for running the Test Script

Valgrind can be disabled with VALGRINDFLAGS=DISABLE
one may define TESTMODE containing any one of the following strings:
- FAST only run tests which failed recently
- FIRSTFAIL abort the tests at the first failure
the variable TESTSUITES may contain a list of string which are used to select which tests are run.
If not given, all available tests are run.

Examples:

running a very fast check while hacking::

(cd target; TESTSUITES=41 VALGRINDFLAGS=DISABLE TESTMODE=FAST+FIRSTFAIL ../tests/test.sh)

invoking the buildsystem, rebuilding if necessary, then invoking just the steam-layer test collections::
```
scons VALGRIND=false TESTSUITES=4 check
```
Running the testsuite with everything enabled is just::
```
scons check
```

Writing test collection definitions

The definitions for test collections usable with test.sh are written in files named name.tests in the tests/ directory dir, where is a number defining the order of the various test files. Of course, “name” should be a descriptive name about what is going to be tested. Each test collection may invoke only a single binary — yet it may define numerous test cases, each invoking this binary while supplementing different arguments. Combined with the ability of our test runner executables to invoke individual test classes, this allows for fine grained test case specifications.

In a *.tests file the following things should be defined:

TESTING <description> <binary> sets the program binary to be tested to the command line <binary>, while <description> should be a string which is displayed as header before running following tests
TEST <description> [optargs] <<END invokes the previously set binary, optionally with additional arguments. <description> is displayed when running this individual test case. A detailed test spec must follow this command and be terminated with END on a single line.
these detailed test specs can contain following statements:
- in: <line> sends <line> to stdin of the test program
- out: <regexp> matches the STDOUT of the test program with the reguar expression
- out-lit: <line> expects <line> to appear literally in the stdout of the test program
- err: <regexp>
- err-lit: <line> similar for STDERR
- return: <status> expect <status> as exit status of the program
if no out: or err: is given, stdout and stderr are not considered as part of the test.
if no return: is given, then 0 is expected.
a detailed log of the test collection invocation is saved in ,testlog

Numbering of test collection definitions

We establish some numbering conventions to ensure running simpler tests prior to more complex ones. Likewise, more advanced integration features should be tested after the application basics.

The currently employed numbering scheme is as follows

00	The test system itself
01	Infrastructure, package consistency
10	Basic support library functionality
20	Higher level support library services
30	Vault Layer Unit tests
40	Steam Layer Unit tests
50	Stage Layer Unit tests (UI binding, Scripting)
60	Component integration tests
70	Functionality tests on the complete application
80	Reported bugs which can be expressed in a test case
90	Optional tests, example code

git://git.lumiera.org/LUMIERA →Gitweb	TRAC · timeline · roadmap
master · gui · proc · back · dok · web	recent · stalled · core-work · non-code
Builddrone · log	API Documentation (Doxygen)	Impressum