testr for openstack

testr for openstack

1 介绍 testr

testr is a test runner and is part of testrepository. testrepository has excellent documentation and a very useful manual. This wiki page will try and condense much of the info found there. testr works by using your test runner to list all of your tests, it takes this list and partitions it into a number of partitions matching the number of CPUs available on the current machine, then forks that number test runners giving each one their own partition of the test list. The test runners are expected to speak subunit back to testr so that testr can keep track of test successes and failures along with other statistics.

testr能够并行地运行用例 (这样更快) 并且能保证输出日志的正确.

2 使用 testr

Nova, neutron和 keystone 是目前正在使用testr的三个openstack项目 (nova移植比较困难). 看一个工程是否使用testr, 在该工程根目录下查看tox.ini文件里的testenv部分的命令设置。

你可以通过已经存在的封装、tox 和run_tests.sh来使用testr
命令如下：

tox -epy27 ortox -epy26ortox -ecover orrun_tests.sh

3 写testr用例

已经确定nose constructs并不能很好地运行testr. 所有的nova单元测试都将用testr来执行. 你可以通过一个正则表达式指定你想要运行的用例 (正则表达式就是你用例的完整路径，例：nova.tests.test_something.TestSomething.test_this_one_unit):

tox -epy27 -- test_name_regex orrun_tests.sh test_name_regex

testr 当你直接运行的时候也很有用. 首先导入你的测试环境

source .tox/py27/bin/activate

导入环境后便能直接运行testr命令了

testr helptestr run --paralleltestr run --parallel test_name_regex

在你测试过后你就能看到那些失败的用例，然后可以通过以下命令只运行它们

testr failingtestr run –failing

3.1 测试日志

你会发现testr把日志保存在.testrepository/$TEST_RUN_NUMBER，这个变量是你运行的数字代号。These logs are subunit streams and are used to determine which tests are failing when you run testr failing. There are a couple neat consequences of this, you can pass these logs around to collaborate on a set of failing tests, it is really easy to write scripts that parse and transform the logs (see python-subunit and its subunit parser), and you can use the logs to help reproduce errors by using the log feed the testr test list fed to the test runners.

3.2 重现失败用例

一种重现失败测试的办法就是按照你第一次执行它们失败时的顺序再执行一遍。为了得到这个测试列表我们首先需要知道那些运行失败的例子子进程的worker name.
在.testrepository/$LAST_TEST_RUN_NUMBER里面可以看到失败的用例，在tags区间里你应该可以看到如下内容
tags: worker-3

With this worker name we can extract the list of tests that ran in that test run on that worker.

source .tox/py27/bin/activatetestr last --subunit | subunit-filter -s --xfail --with-tag=worker-3 | subunit-ls > slave-3.list

Using this test list we can run that set of tests in the same order that caused the failure with:

source .tox/py27/bin/activatetestr run --load-list=slave-3.list

testr also comes with a really fancy automated test bisection feature that will try to determine the minimal set of tests required to reproduce failures that result when tests interfere with each other. To use this feature run

source .tox/py27/bin/activatetestr run --analyze-isolation

after you have had a failed test run.

4 Debugging (pdb) Tests

debugging tests requires use of testtools.run. the bug listing here explains why direct pdb support does not work. There is a simple process to get use of pdb within tests. First, generate a list of tests to run

source .tox/py27/bin/activatetestr list-tests test_name_regex > my-list

Note: test_name_regex is the same as previously mentioned in the document Now pipe it through testtools.run

python -m testtools.run discover --load-list my-list

And wherever you set your pdb.set_trace() will break into the debugger

5 FAQ

怎么执行一个或多个用例
To limit the tests that are run, provide tox with a suitably narrow regex to limit test discovery:

Run only the tests in the TestAlarms and TestDeprecatedPipeline classes
tox -epy27 – ‘(TestAlarms|TestDeprecatedPipeline)’

To run one test:

source .tox/py27/bin/activate
python -m testtools.run

怎么在一次失败后终止用例执行
Sometimes you want to run all the tests but exit after the first failure (and not wait until then end). In projects that use subunit.run as the testrunner at the bottom of the stack this is possible by using testr directly and skipping tox:

warm up the virtualenv

source .tox/py27/bin/activate

start the tests with -f

testr run -- -f

In some environments (for example Ceilometer) it is necessary to establish the testing environment:

warm up the virtualenv

source .tox/py27/bin/activate

start the tests with -f in the right environment

./setup-test-env.sh testr run -- -f

当看到”The test run didn’t actually run any tests“输出显示时该怎么做
This can happen when an error is encountered during test listing, such as an import error.

Non-zero exit code (2) from test listing.
error: testr failed (3)
The test run didn’t actually run any tests

To see a trace, list the tests:

source .tox/py27/bin/activatetestr list-tests

翻译自：https://wiki.openstack.org/wiki/Testr 2016.03.30