floraachy/pytest2
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

some finalizing docs bit, regen plugin docs

Browse Source 
--HG--
branch : trunk
This commit is contained in:
holger krekel
2010-01-18 16:18:59 +01:00
parent 1e76202b65
commit 49f7972d48
8 changed files with 188 additions and 219 deletions
Download Patch File Download Diff File Expand all files Collapse all files

132
doc/test/dist.txt
View File

@@ -1,132 +0,0 @@
.. _`distribute tests across machines`:
===================
Distributed testing
===================
``py.test`` can ad-hoc distribute test runs to multiple CPUs or remote
machines. This allows to speed up development or to use special resources
of remote machines. Before running tests remotely, ``py.test`` efficiently
synchronizes your program source code to the remote place. All test results
are reported back and displayed to your local test session. You may
specify different Python versions and interpreters.
**Requirements**: you need to install the `execnet`_ package
(at least version 1.0.0b4) to perform distributed test runs.
**NOTE**: Version 1.1.x is not able to distribute tests across Python3/Python2 barriers.
Speed up test runs by sending tests to multiple CPUs
----------------------------------------------------------
To send tests to multiple CPUs, type::
py.test -n NUM
Especially for longer running tests or tests requiring
a lot of IO this can lead to considerable speed ups.
Running tests in a Python subprocess
----------------------------------------
To instantiate a python2.4 sub process and send tests to it, you may type::
py.test -d --tx popen//python=python2.4
This will start a subprocess which is run with the "python2.4"
Python interpreter, found in your system binary lookup path.
If you prefix the --tx option value like this::
--tx 3*popen//python=python2.4
then three subprocesses would be created and tests
will be load-balanced across these three processes.
Sending tests to remote SSH accounts
-----------------------------------------------
Suppose you have a package ``mypkg`` which contains some
tests that you can successfully run locally. And you
have a ssh-reachable machine ``myhost``. Then
you can ad-hoc distribute your tests by typing::
py.test -d --tx ssh=myhostpopen --rsyncdir mypkg mypkg
This will synchronize your ``mypkg`` package directory
to an remote ssh account and then locally collect tests
and send them to remote places for execution.
You can specify multiple ``--rsyncdir`` directories
to be sent to the remote side.
**NOTE:** For py.test to collect and send tests correctly
you not only need to make sure all code and tests
directories are rsynced, but that any test (sub) directory
also has an ``__init__.py`` file because internally
py.test references tests as a fully qualified python
module path. **You will otherwise get strange errors**
during setup of the remote side.
Sending tests to remote Socket Servers
----------------------------------------
Download the single-module `socketserver.py`_ Python program
and run it like this::
python socketserver.py
It will tell you that it starts listening on the default
port. You can now on your home machine specify this
new socket host with something like this::
py.test -d --tx socket=192.168.1.102:8888 --rsyncdir mypkg mypkg
.. _`atonce`:
Running tests on many platforms at once
-------------------------------------------------------------
The basic command to run tests on multiple platforms is::
py.test --dist=each --tx=spec1 --tx=spec2
If you specify a windows host, an OSX host and a Linux
environment this command will send each tests to all
platforms - and report back failures from all platforms
at once. The specifications strings use the `xspec syntax`_.
.. _`xspec syntax`: http://codespeak.net/execnet/trunk/basics.html#xspec
.. _`socketserver.py`: http://codespeak.net/svn/py/dist/py/execnet/script/socketserver.py
.. _`execnet`: http://codespeak.net/execnet
Specifying test exec environments in a conftest.py
-------------------------------------------------------------
Instead of specifying command line options, you can
put options values in a ``conftest.py`` file like this::
pytest_option_tx = ['ssh=myhost//python=python2.5', 'popen//python=python2.5']
pytest_option_dist = True
Any commandline ``--tx`` specifictions will add to the list of available execution
environments.
Specifying "rsync" dirs in a conftest.py
-------------------------------------------------------------
In your ``mypkg/conftest.py`` you may specify directories to synchronise
or to exclude::
rsyncdirs = ['.', '../plugins']
rsyncignore = ['_cache']
These directory specifications are relative to the directory
where the ``conftest.py`` is found.

105
doc/test/features.txt
View File

@@ -9,22 +9,40 @@ py.test feature overview
mature command line testing tool
====================================================
py.test is a command line tool to collect and run automated tests. It
runs well on Linux, Windows and OSX Python 2.4 through to 2.6 versions.
It can distribute a single test run to multiple machines. It is used in
many projects, ranging from running 10 thousands of tests integrated
with buildbot to a few inlined tests on a command line script.
py.test is a command line tool to collect, run and report about automated tests. It runs well on Linux, Windows and OSX and on Python 2.4 through to 3.1 versions.
It is used in many projects, ranging from running 10 thousands of tests
to a few inlined tests on a command line script. As of version 1.2 you can also
generate a no-dependency py.test-equivalent standalone script that you
can distribute along with your application.
.. _`autocollect`:
extensive easy plugin system
======================================================
automatically collects and executes tests
===============================================
.. _`suprisingly easy`: http://bruynooghe.blogspot.com/2009/12/skipping-slow-test-by-default-in-pytest.html
py.test discovers tests automatically by looking at
specified directories and its files for common
naming patterns. As ``py.test`` operates as a separate
cmdline tool you can easily have a command line utility and
some tests in the same file.
py.test delegates almost all aspects of its operation to plugins_.
It is `suprisingly easy`_ to add command line options or
do other kind of add-ons and customizations. This can
be done per-project or by distributing a global plugin.
One can can thus modify or add aspects for purposes such as:
* reporting extensions
* customizing collection and execution of tests
* running and managing non-python tests
* managing domain-specific test state setup
* adding non-python tests into the run, e.g. driven by data files
.. _`plugins`: plugin/index.html
distributing tests to your CPUs and SSH accounts
==========================================================
.. _`pytest-xdist`: plugin/xdist.html
Through the use of the separately released `pytest-xdist`_ plugin you
can seemlessly distribute runs to multiple CPUs or remote computers
through SSH and sockets. This plugin also offers a ``--looponfailing``
mode which will continously re-run only failing tests in a subprocess.
supports several testing practises and methods
==================================================================
@@ -40,9 +58,20 @@ with figleaf`_ or `Javasript unit- and functional testing`_.
.. _`Javasript unit- and functional testing`: plugin/oejskit.html
.. _`coverage testing with figleaf`: plugin/figleaf.html
integrates well with CI systems
====================================================
py.test can produce JUnitXML style output as well as formatted
"resultlog" files that can be postprocessed by Continous Integration
systems such as Hudson or Buildbot easily. It also provides command
line options to control test configuration lookup behaviour or ignoring
certain tests or directories.
no-boilerplate test functions with Python
===================================================
.. _`autocollect`:
automatic Python test discovery
------------------------------------
@@ -53,13 +82,18 @@ filename are inspected for finding tests:
* classes with a leading ``Test`` name and ``test`` prefixed methods.
* ``unittest.TestCase`` subclasses
test functions can run with different argument sets
-----------------------------------------------------------
parametrizing test functions and advanced functional testing
--------------------------------------------------------------
py.test offers the unique `funcargs mechanism`_ for setting up
and passing project-specific objects to Python test functions.
Test Parametrization happens by triggering a call to the same test
functions with different argument values.
function with different argument values. For doing fixtures
using the funcarg mechanism makes your test and setup code
more efficient and more readable. This is especially true
for functional tests which might depend on command line
options and a setup that needs to be shared across
a whole test run.
per-test capturing of output, including subprocesses
----------------------------------------------------
@@ -137,30 +171,6 @@ can make use of this feature.
.. _`xUnit style setup`: xunit_setup.html
.. _`pytest_nose`: plugin/nose.html
load-balance test runs to multiple CPUs
========================================
For large test suites you can distribute your
tests to multiple CPUs by issuing for example::
py.test -n 3
Read more on `distributed testing`_.
.. _`distributed testing`: dist.html
ad-hoc run tests cross-platform
==================================================
py.test supports the sending of tests to
remote ssh-accounts, socket servers.
It can `ad-hoc run your test on multiple
platforms one a single test run`. Ad-hoc
means that there are **no installation
requirements whatsoever** on the remote side.
.. _`ad-hoc run your test on multiple platforms one a single test run`: dist.html#atonce
advanced test selection and running modes
=========================================================
@@ -202,21 +212,6 @@ plugin for more information.
.. _`pytest_keyword`: plugin/mark.html
easy to extend
=========================================
py.test has advanced `extension mechanisms`_
with a growing `list of default plugins`_.
One can can easily modify or add aspects for for
purposes such as:
* reporting extensions
* customizing collection and execution of tests
* running and managing non-python tests
* managing domain-specific test state setup
.. _`list of default plugins`: plugin/index.html
.. _`extension mechanisms`: customize.html#extensions
.. _`reStructured Text`: http://docutils.sourceforge.net
.. _`Python debugger`: http://docs.python.org/lib/module-pdb.html

1
doc/test/index.txt
View File

@@ -24,6 +24,5 @@ changelog_: history of changes covering last releases
.. _features: features.html
.. _funcargs: funcargs.html
.. _customize: customize.html
.. _`distributed testing`: dist.html

30
doc/test/plugin/index.txt
View File

@@ -8,7 +8,7 @@ mark_ generic mechanism for marking python functions.
pdb_ interactive debugging with the Python Debugger.
figleaf_ report test coverage using the 'figleaf' package.
figleaf_ (external) report test coverage using the 'figleaf' package.
coverage_ (external) for testing with Ned's coverage module
@@ -21,34 +21,26 @@ recwarn_ helpers for asserting deprecation and other warnings.
tmpdir_ provide temporary directories to test functions.
other testing domains, misc
===========================
distributed testing, CI and deployment
======================================
oejskit_ (external) run javascript tests in real life browsers
django_ (external) for testing django applications
xdist_ loop on failing tests, distribute test runs to CPUs and hosts.
genscript_ generate standalone test script to be distributed along with an application.
reporting and failure logging
=============================
xdist_ (external) loop on failing tests, distribute test runs to CPUs and hosts.
pastebin_ submit failure or test session information to a pastebin service.
junitxml_ logging of test results in JUnit-XML format, for use with Hudson
xmlresult_ (external) for generating xml reports and CruiseControl integration
resultlog_ non-xml machine-readable logging of test results.
terminal_ Implements terminal reporting of the full testing process.
genscript_ generate standalone test script to be distributed along with an application.
other testing conventions
=========================
testing domains and conventions
===============================
oejskit_ (external) run javascript tests in real life browsers
django_ (external) for testing django applications
unittest_ automatically discover and run traditional "unittest.py" style tests.

5
doc/test/plugin/links.txt
View File

@@ -1,5 +1,4 @@
.. _`helpconfig`: helpconfig.html
.. _`terminal`: terminal.html
.. _`pytest_recwarn.py`: http://bitbucket.org/hpk42/py-trunk/raw/1.2.0/py/plugin/pytest_recwarn.py
.. _`unittest`: unittest.html
.. _`pytest_monkeypatch.py`: http://bitbucket.org/hpk42/py-trunk/raw/1.2.0/py/plugin/pytest_monkeypatch.py
@@ -15,7 +14,6 @@
.. _`pytest_nose.py`: http://bitbucket.org/hpk42/py-trunk/raw/1.2.0/py/plugin/pytest_nose.py
.. _`pytest_restdoc.py`: http://bitbucket.org/hpk42/py-trunk/raw/1.2.0/py/plugin/pytest_restdoc.py
.. _`restdoc`: restdoc.html
.. _`xdist`: xdist.html
.. _`pytest_pastebin.py`: http://bitbucket.org/hpk42/py-trunk/raw/1.2.0/py/plugin/pytest_pastebin.py
.. _`pytest_tmpdir.py`: http://bitbucket.org/hpk42/py-trunk/raw/1.2.0/py/plugin/pytest_tmpdir.py
.. _`pytest_figleaf.py`: http://bitbucket.org/hpk42/py-trunk/raw/1.2.0/py/plugin/pytest_figleaf.py
@@ -33,15 +31,14 @@
.. _`figleaf`: figleaf.html
.. _`customize`: ../customize.html
.. _`hooklog`: hooklog.html
.. _`pytest_terminal.py`: http://bitbucket.org/hpk42/py-trunk/raw/1.2.0/py/plugin/pytest_terminal.py
.. _`recwarn`: recwarn.html
.. _`xdist`: xdist.html
.. _`pytest_pdb.py`: http://bitbucket.org/hpk42/py-trunk/raw/1.2.0/py/plugin/pytest_pdb.py
.. _`monkeypatch`: monkeypatch.html
.. _`coverage`: coverage.html
.. _`resultlog`: resultlog.html
.. _`pytest_junitxml.py`: http://bitbucket.org/hpk42/py-trunk/raw/1.2.0/py/plugin/pytest_junitxml.py
.. _`django`: django.html
.. _`xmlresult`: xmlresult.html
.. _`pytest_unittest.py`: http://bitbucket.org/hpk42/py-trunk/raw/1.2.0/py/plugin/pytest_unittest.py
.. _`nose`: nose.html
.. _`pytest_resultlog.py`: http://bitbucket.org/hpk42/py-trunk/raw/1.2.0/py/plugin/pytest_resultlog.py

2
doc/test/plugin/xdist.txt
View File

@@ -26,7 +26,7 @@ program source code to the remote place. All test results
are reported back and displayed to your local test session.
You may specify different Python versions and interpreters.
.. _`pytest-xdist`: http://pytest.org/plugin/xdist.html
.. _`pytest-xdist`: http://pypi.python.org/pypi/pytest-xdist
Usage examples
---------------------