Skip to content

rerunning

Jump to bottom
Richard R. Drake edited this page Jun 8, 2022 · 1 revision

Selectively rerunning tests

When vvtest is run for the first time in a directory, a subdirectory is created whose name starts with "TestResults." and then is followed by the platform name, then followed by -o option values. This is the directory tree where all the test executions take place. Also in the test execution directory, is a cache file and files that contain the list of tests and the results of running the tests.

When vvtest is run again, it recognizes the existing test execution directory and loads the information it contains. That is, it "remembers" the previous command and the state of each of the test executions. This allows you, for example, to get the results of a previous test run by executing vvtest -i.

Using -R to rerun tests

By default, a subsequent vvtest run will only run tests that have not been run or that did not finish. You can cause all tests to be rerun by adding the -R option to the command line. For example, consider if the test file "mytest.vvt" existed in the current directory and resulted in a single test.

s968057% ls
mytest.vvt
s968057% vvtest
<snip>
==================================================
mytest               Exit    pass     1s   07/10 08:18:42 TestResults.Darwin/mytest
==================================================
Summary: 1 pass, 0 timeout, 0 diff, 0 fail, 0 notrun, 0 notdone

Finish date: Sun Jul 10 08:18:43 2016 (elapsed time 1s)
Test directory: TestResults.Darwin
s968057%
s968057% vvtest

--------- no tests to run -----------

s968057%
s968057% vvtest -R
<snip>
==================================================
mytest               Exit    pass     1s   07/10 08:19:02 TestResults.Darwin/mytest
==================================================
Summary: 1 pass, 0 timeout, 0 diff, 0 fail, 0 notrun, 0 notdone

Finish date: Sun Jul 10 08:19:03 2016 (elapsed time 1s)
Test directory: TestResults.Darwin

Note how the second execution has "no tests to run", but adding the -R option causes the test to run again.

Using -k with results keywords

There are a handful of implicit keywords, called "results keywords", that every test knows about. They are

  • pass: the test was run and exited cleanly (a zero exit status)
  • diff: the test was run and exited with a diff exit status
  • fail: the test was run and exited with a non-zero exit status
  • timeout: the test was run and was killed due to running out of time
  • notrun: the test was not launched
  • notdone: the test was launched but did not finish

These keywords can be used in -k option arguments. For example, to run all tests again that failed or timed out, do this

$ vvtest -k fail/timeout

To rerun tests that did not pass or diff, do this

$ vvtest -K pass -K diff

And to run all tests that did not finish or did not run, do this

$ vvtest -k notdone/notrun

Note that this last command has the same effect as running without the -R option. In fact, the -R option does nothing more than prevent the addition of -k notdone/notrun on the command line. Similarly, if any -k or -K option expression contains a results keyword, then -k notdone/notrun is not added to the command line.

Finally, note that results keyword specifications can be combined with non-results keyword specifications. For example, this would run all tests that diffed and that had the "mhd" keyword

$ vvtest -k diff -k mhd

Rerunning tests selected by parameter or platform

The -R option to rerun tests can be combined with the -p and -P options in order to select tests to rerun by their parameter names and values. For example, this would rerun all tests that had the "np" parameter name defined and whose value is less than 8:

$ vvtest -R -p 'np<8'

Similarly, the platform filtering options -x and -X can be used with -R to rerun tests selected by platform.

See the chapter "Filtering and selecting tests" for more information on the filtering/selection mechanisms.

Using the wipe option to rerun tests

The -w option means remove all files in the test results directory before running. The behavior then is as if no tests had run previously.

Rerunning tests by directory

Tests are run based on the current working directory as well. You can change directory to be somewhere inside the test results tree, then rerun vvtest. The tests that are considered for running are only those in the current working directory or below. For example,

s968057% ls -R
subdir_1 subdir_2
./subdir_1:
test1.vvt
./subdir_2:
test2.vvt

s968057% vvtest
Running these tests:
<snip>
==================================================
test1                Exit    pass     1s   08/06 06:41:17 TestResults.Darwin/subdir_1/test1
test2                Exit    pass     1s   08/06 06:41:17 TestResults.Darwin/subdir_2/test2
==================================================
Summary: 2 pass, 0 timeout, 0 diff, 0 fail, 0 notrun, 0 notdone

Finish date: Sat Aug  6 06:41:18 2016 (elapsed time 1s)
Test directory: TestResults.Darwin

s968057% cd TestResults.Darwin/subdir_2
s968057% vvtest

--------- no tests to run -----------

s968057% vvtest -R
Running these tests:
<snip>
==================================================
test2                Exit    pass     1s   08/06 06:42:29 test2
==================================================
Summary: 1 pass, 0 timeout, 0 diff, 0 fail, 0 notrun, 0 notdone

Finish date: Sat Aug  6 06:42:30 2016 (elapsed time 1s)

Note that only "test2" ran when in the subdir_2 subdirectory.

Lastly, filtering command line options can be applied when in a subdirectory as well, such as -p and -k.

Clone this wiki locally