From c4fe18608d98a255a07ab23b87a9364eec654912 Mon Sep 17 00:00:00 2001 From: hpk Date: Mon, 23 Mar 2009 11:01:15 +0100 Subject: [PATCH] [svn r63222] refactor test-features doc, make references to others --HG-- branch : trunk --- py/doc/download.txt | 2 - py/doc/test-dist.txt | 2 + py/doc/test-features.txt | 305 ++++++++++++++++++++++----------------- 3 files changed, 174 insertions(+), 135 deletions(-) diff --git a/py/doc/download.txt b/py/doc/download.txt index 8127ff6d5..d94bb8cea 100644 --- a/py/doc/download.txt +++ b/py/doc/download.txt @@ -1,5 +1,3 @@ - - "easy_install py" =================================================== diff --git a/py/doc/test-dist.txt b/py/doc/test-dist.txt index 83b29040c..1b32df428 100644 --- a/py/doc/test-dist.txt +++ b/py/doc/test-dist.txt @@ -74,6 +74,8 @@ new socket host with something like this:: py.test -d --tx socket=192.168.1.102:8888 --rsyncdir mypkg mypkg +.. _`atonce`: + Running tests on many platforms at once ------------------------------------------------------------- diff --git a/py/doc/test-features.txt b/py/doc/test-features.txt index e5ef9297e..ec615609f 100644 --- a/py/doc/test-features.txt +++ b/py/doc/test-features.txt @@ -1,137 +1,59 @@ -automatic collection of tests on all levels -------------------------------------------- +.. contents:: Basic features + :depth: 1 -The automated test collection process walks the current -directory (or the directory given as a command line argument) -and all its subdirectories and collects python modules with a -leading ``test_`` or trailing ``_test`` filename. From each -test module every function with a leading ``test_`` or class with -a leading ``Test`` name is collected. The collecting process can -be customized at directory, module or class level. (see -`collection process`_ for some implementation details). +py.test: cross-project general testing tool +================================================== + +py.test is a standalone-tool that collects and runs tests for +your Python application and modules. py.test works across +linux, windows and osx and on Python 2.3 - Python 2.6. + +It aims to support *unit-tests* and *functional tests* written +in Python and is used in projects that run more than 10000 +tests regularly. + +py.test presents a clean and powerful command line interface +and strives to generally make testing a fun effort. + +automatically collects and executes tests +=============================================== + +py.test discovers tests automatically by inspect specified +directories or files. By default, it collects all python +modules a leading ``test_`` or trailing ``_test`` filename. +From each test module every function with a leading ``test_`` +or class with a leading ``Test`` name is collected. .. _`generative tests`: .. _`collection process`: impl-test.html#collection-process -assert with the ``assert`` statement ------------------------------------- +load-balance tests to multiple CPUs +=================================== -``py.test`` allows to use the standard python -``assert statement`` for verifying expectations -and values in Python tests. For example, you can -write the following in your tests:: +For large test suites you can distribute your +tests to multiple CPUs by issuing for example:: - assert hasattr(x, 'attribute') + py.test -n 3 -to state that your object has a certain ``attribute``. In case this -assertion fails you will see the value of ``x``. Intermediate -values are computed by executing the assert expression a second time. -If you execute code with side effects, e.g. read from a file like this:: +Read more on `distributed testing`_. - assert f.read() != '...' +.. _`distributed testing`: test-dist.html -then you may get a warning from pytest if that assertions -first failed and then succeeded. +Distribute tests across machines +=================================== -asserting expected exceptions ----------------------------------------------- +py.test supports the sending of tests to +remote ssh-accounts or socket servers. +It can `ad-hoc run your test on multiple +platforms one a single test run`. Ad-hoc +means that there are **no installation +requirements whatsoever** on the remote side. -In order to write assertions about exceptions, you use -one of two forms:: - - py.test.raises(Exception, func, *args, **kwargs) - py.test.raises(Exception, "func(*args, **kwargs)") - -both of which execute the given function with args and kwargs and -asserts that the given ``Exception`` is raised. The reporter will -provide you with helpful output in case of failures such as *no -exception* or *wrong exception*. - -dynamically skipping tests ----------------------------------------- - -If you want to skip tests you can use ``py.test.skip`` within -test or setup functions. Example:: - - py.test.skip("message") - -You can also use a helper to skip on a failing import:: - - docutils = py.test.importorskip("docutils") - -or to skip if a library does not have the right version:: - - docutils = py.test.importorskip("docutils", minversion="0.3") - -The version will be read from the module's ``__version__`` attribute. - - -generative tests: yielding more tests -------------------------------------- - -*Generative tests* are test methods that are *generator functions* which -``yield`` callables and their arguments. This is most useful for running a -test function multiple times against different parameters. Example:: - - def test_generative(): - for x in (42,17,49): - yield check, x - - def check(arg): - assert arg % 7 == 0 # second generated tests fails! - -Note that ``test_generative()`` will cause three tests -to get run, notably ``check(42)``, ``check(17)`` and ``check(49)`` -of which the middle one will obviously fail. - -To make it easier to distinguish the generated tests it is possible to specify an explicit name for them, like for example:: - - def test_generative(): - for x in (42,17,49): - yield "case %d" % x, check, x - - -.. _`selection by keyword`: - -selecting/unselecting tests by keyword ---------------------------------------------- - -Pytest's keyword mechanism provides a powerful way to -group and selectively run tests in your test code base. -You can selectively run tests by specifiying a keyword -on the command line. Examples: - - py.test -k test_simple - py.test -k "-test_simple" - -will run all tests matching (or not matching) the -"test_simple" keyword. Note that you need to quote -the keyword if "-" is recognized as an indicator -for a commandline option. Lastly, you may use - - py.test. -k "test_simple:" - -which will run all tests after the expression has *matched once*, i.e. -all tests that are seen after a test that matches the "test_simple" -keyword. - -By default, all filename parts and -class/function names of a test function are put into the set -of keywords for a given test. You may specify additional -kewords like this:: - - @py.test.mark(webtest=True) - def test_send_http(): - ... - -testing with multiple python versions / executables ---------------------------------------------------- - -With ``--tx EXECUTABLE`` you can specify a python -executable (e.g. ``python2.2``) with which the tests -will be executed. +.. _`ad-hoc run your test on multiple platforms one a single test run`: test-dist.html#atonce +extensive debugging support +=================================== testing starts immediately -------------------------- @@ -174,6 +96,40 @@ this prevents more advanced usages: running tests distributedly or selectively, or in "looponfailing" mode, will cause them to run in random order. +assert with the ``assert`` statement +---------------------------------------- + +``py.test`` allows to use the standard python +``assert statement`` for verifying expectations +and values in Python tests. For example, you can +write the following in your tests:: + + assert hasattr(x, 'attribute') + +to state that your object has a certain ``attribute``. In case this +assertion fails you will see the value of ``x``. Intermediate +values are computed by executing the assert expression a second time. +If you execute code with side effects, e.g. read from a file like this:: + + assert f.read() != '...' + +then you may get a warning from pytest if that assertions +first failed and then succeeded. + +asserting expected exceptions +---------------------------------------- + +In order to write assertions about exceptions, you use +one of two forms:: + + py.test.raises(Exception, func, *args, **kwargs) + py.test.raises(Exception, "func(*args, **kwargs)") + +both of which execute the given function with args and kwargs and +asserts that the given ``Exception`` is raised. The reporter will +provide you with helpful output in case of failures such as *no +exception* or *wrong exception*. + useful tracebacks, recursion detection -------------------------------------- @@ -182,7 +138,7 @@ failure. Try:: py.test py/doc/example/pytest/failure_demo.py -to see a variety of 17 tracebacks, each tailored to a different +to see a variety of tracebacks, each representing a different failure situation. ``py.test`` uses the same order for presenting tracebacks as Python @@ -206,6 +162,68 @@ them be found by the test runner. Besides being easier, it also allows you to write test classes that subclass from application level classes. +testing for deprecated APIs +------------------------------ + +In your tests you can use ``py.test.deprecated_call(func, *args, **kwargs)`` +to test that a particular function call triggers a DeprecationWarning. +This is useful for testing phasing out of old APIs in your projects. + + +advanced test selection / skipping +========================================================= + +dynamically skipping tests +------------------------------- + +If you want to skip tests you can use ``py.test.skip`` within +test or setup functions. Example:: + + py.test.skip("message") + +You can also use a helper to skip on a failing import:: + + docutils = py.test.importorskip("docutils") + +or to skip if a library does not have the right version:: + + docutils = py.test.importorskip("docutils", minversion="0.3") + +The version will be read from the module's ``__version__`` attribute. + +.. _`selection by keyword`: + +selecting/unselecting tests by keyword +--------------------------------------------- + +Pytest's keyword mechanism provides a powerful way to +group and selectively run tests in your test code base. +You can selectively run tests by specifiying a keyword +on the command line. Examples:: + + py.test -k test_simple + py.test -k "-test_simple" + +will run all tests matching (or not matching) the +"test_simple" keyword. Note that you need to quote +the keyword if "-" is recognized as an indicator +for a commandline option. Lastly, you may use:: + + py.test. -k "test_simple:" + +which will run all tests after the expression has *matched once*, i.e. +all tests that are seen after a test that matches the "test_simple" +keyword. + +By default, all filename parts and +class/function names of a test function are put into the set +of keywords for a given test. You may specify additional +kewords like this:: + + @py.test.mark(webtest=True) + def test_send_http(): + ... + disabling a test class ---------------------- @@ -219,22 +237,43 @@ For example, in order to avoid running some tests on Win32:: def test_xxx(self): ... -testing for deprecated APIs ------------------------------- +generative tests: yielding parametrized tests +==================================================== -In your tests you can use ``py.test.deprecated_call(func, *args, **kwargs)`` -to test that a particular function call triggers a DeprecationWarning. -This is useful for testing phasing out of old APIs in your projects. +*Generative tests* are test methods that are *generator functions* which +``yield`` callables and their arguments. This is most useful for running a +test function multiple times against different parameters. Example:: -doctest support -------------------- + def test_generative(): + for x in (42,17,49): + yield check, x + + def check(arg): + assert arg % 7 == 0 # second generated tests fails! -If you want to integrate doctests, ``py.test`` now by default -picks up files matching the ``test_*.txt`` or ``*_test.txt`` -patterns and processes them as text files containing doctests. -This is an experimental feature and likely to change -its implementation. +Note that ``test_generative()`` will cause three tests +to get run, notably ``check(42)``, ``check(17)`` and ``check(49)`` +of which the middle one will obviously fail. +To make it easier to distinguish the generated tests it is possible to specify an explicit name for them, like for example:: + + def test_generative(): + for x in (42,17,49): + yield "case %d" % x, check, x + +extensible plugin system +========================================= + +py.test itself consists of many plugins +and you can easily write new `py.test plugins`_ +for these purposes: + +* reporting extensions +* customizing collection and run of tests +* running non-python tests +* managing test state setup + +.. _`py.test plugins`: test-plugins.html .. _`reStructured Text`: http://docutils.sourceforge.net .. _`Python debugger`: http://docs.python.org/lib/module-pdb.html