335 lines
12 KiB
Plaintext
335 lines
12 KiB
Plaintext
|
|
V4: Creating and working with parametrized resources
|
|
===============================================================
|
|
|
|
**Target audience**: Reading this document requires basic knowledge of
|
|
python testing, xUnit setup methods and the basic pytest funcarg mechanism,
|
|
see http://pytest.org/latest/funcargs.html
|
|
|
|
**Abstract**: pytest-2.X provides yet more powerful and flexible
|
|
fixture machinery by introducing:
|
|
|
|
* a new ``@pytest.mark.funcarg`` marker to define funcarg factories and their
|
|
scoping and parametrization. No special ``pytest_funcarg__`` naming there.
|
|
|
|
* a new ``@pytest.mark.setup`` marker to define setup functions and their
|
|
scoping.
|
|
|
|
* directly use funcargs through funcarg factory signatures
|
|
|
|
Both funcarg factories and setup functions can be defined in test modules,
|
|
classes, conftest.py files and installed plugins.
|
|
|
|
The introduction of these two markers lifts several prior limitations
|
|
and allows to easily define and implement complex testing scenarios.
|
|
|
|
Nonwithstanding these extensions, already existing test suites and plugins
|
|
written to work for previous pytest versions shall run unmodified.
|
|
|
|
|
|
**Changes**: This V4 draft is based on incorporating and thinking about
|
|
feedback on previous versions provided by Floris Bruynooghe, Carl Meyer,
|
|
Ronny Pfannschmidt and Samuele Pedroni. It remains as draft
|
|
documentation, pending further refinements and changes according to
|
|
implementation or backward compatibility issues. The main changes are:
|
|
|
|
* Collapse funcarg factory decorators into a single "@funcarg" one.
|
|
You can specify scopes and params with it. When using the decorator
|
|
the "pytest_funcarg__" prefix becomes optional.
|
|
|
|
* funcarg factories can now use funcargs themselves
|
|
|
|
* Drop setup/directory scope from this draft
|
|
|
|
* introduce a new @setup decorator similar to the @funcarg one
|
|
except that setup-markers cannot define parametriation themselves.
|
|
Instead they can easily depend on a parametrized funcarg (which
|
|
must not be visible at test function signatures).
|
|
|
|
* drop consideration of setup_X support for funcargs because
|
|
it is less flexible and probably causes more implementation
|
|
troubles than the current @setup approach which can share
|
|
a lot of logic with the @funcarg one.
|
|
|
|
* tests are grouped by parametrized funcargs
|
|
|
|
.. currentmodule:: _pytest
|
|
|
|
|
|
Shortcomings of the previous pytest_funcarg__ mechanism
|
|
---------------------------------------------------------
|
|
|
|
The previous funcarg mechanism calls a factory each time a
|
|
funcarg for a test function is requested. If a factory wants
|
|
t re-use a resource across different scopes, it often used
|
|
the ``request.cached_setup()`` helper to manage caching of
|
|
resources. Here is a basic example how we could implement
|
|
a per-session Database object::
|
|
|
|
# content of conftest.py
|
|
class Database:
|
|
def __init__(self):
|
|
print ("database instance created")
|
|
def destroy(self):
|
|
print ("database instance destroyed")
|
|
|
|
def pytest_funcarg__db(request):
|
|
return request.cached_setup(setup=DataBase,
|
|
teardown=lambda db: db.destroy,
|
|
scope="session")
|
|
|
|
There are some problems with this approach:
|
|
|
|
1. Scoping resource creation is not straight forward, instead one must
|
|
understand the intricate cached_setup() method mechanics.
|
|
|
|
2. parametrizing the "db" resource is not straight forward:
|
|
you need to apply a "parametrize" decorator or implement a
|
|
:py:func:`~hookspec.pytest_generate_tests` hook
|
|
calling :py:func:`~python.Metafunc.parametrize` which
|
|
performs parametrization at the places where the resource
|
|
is used. Moreover, you need to modify the factory to use an
|
|
``extrakey`` parameter containing ``request.param`` to the
|
|
:py:func:`~python.Request.cached_setup` call.
|
|
|
|
3. there is no way how you can make use of funcarg factories
|
|
in xUnit setup methods.
|
|
|
|
4. A non-parametrized funcarg factory cannot use a parametrized
|
|
funcarg resource if it isn't stated in the test function signature.
|
|
|
|
The following sections address the advances which solve all of these problems.
|
|
|
|
|
|
Direct scoping of funcarg factories
|
|
--------------------------------------------------------
|
|
|
|
.. note:: Implemented
|
|
|
|
Instead of calling cached_setup(), you can decorate your factory
|
|
to state its scope::
|
|
|
|
@pytest.mark.funcarg(scope="session")
|
|
def pytest_funcarg__db(request):
|
|
# factory will only be invoked once per session -
|
|
db = DataBase()
|
|
request.addfinalizer(db.destroy) # destroy when session is finished
|
|
return db
|
|
|
|
This factory implementation does not need to call ``cached_setup()`` anymore
|
|
because it will only be invoked once per session. Moreover, the
|
|
``request.addfinalizer()`` registers a finalizer according to the specified
|
|
resource scope on which the factory function is operating. With this new
|
|
scoping, the still existing ``cached_setup()`` should be much less used
|
|
but will remain for compatibility reasons and for the case where you
|
|
still want to have your factory get called on a per-item basis.
|
|
|
|
|
|
Direct parametrization of funcarg resource factories
|
|
----------------------------------------------------------
|
|
|
|
.. note:: Implemented
|
|
|
|
Previously, funcarg factories could not directly cause parametrization.
|
|
You needed to specify a ``@parametrize`` or implement a ``pytest_generate_tests`` hook to perform parametrization, i.e. calling a test multiple times
|
|
with different value sets. pytest-2.X introduces a decorator for use
|
|
on the factory itself::
|
|
|
|
@pytest.mark.funcarg(params=["mysql", "pg"])
|
|
def pytest_funcarg__db(request):
|
|
...
|
|
|
|
Here the factory will be invoked twice (with the respective "mysql"
|
|
and "pg" values set as ``request.param`` attributes) and and all of
|
|
the tests requiring "db" will run twice as well. The "mysql" and
|
|
"pg" values will also be used for reporting the test-invocation variants.
|
|
|
|
This new way of parametrizing funcarg factories should in many cases
|
|
allow to re-use already written factories because effectively
|
|
``request.param`` are already the parametrization attribute for test
|
|
functions/classes were parametrized via
|
|
:py:func:`~_pytest.python.Metafunc.parametrize(indirect=True)` calls.
|
|
|
|
Of course it's perfectly fine to combine parametrization and scoping::
|
|
|
|
@pytest.mark.funcarg(scope="session", params=["mysql", "pg"])
|
|
def pytest_funcarg__db(request):
|
|
if request.param == "mysql":
|
|
db = MySQL()
|
|
elif request.param == "pg":
|
|
db = PG()
|
|
request.addfinalizer(db.destroy) # destroy when session is finished
|
|
return db
|
|
|
|
This would execute all tests requiring the per-session "db" resource twice,
|
|
receiving the values created by the two respective invocations to the
|
|
factory function.
|
|
|
|
Direct usage of funcargs with funcargs factories
|
|
----------------------------------------------------------
|
|
|
|
.. note:: Implemented.
|
|
|
|
You can now directly use funcargs in funcarg factories. Example::
|
|
|
|
@pytest.mark.funcarg(scope="session")
|
|
def db(request, tmpdir):
|
|
# tmpdir is a session-specific tempdir
|
|
|
|
Apart from convenience it also solves an issue when your factory
|
|
depends on a parametrized funcarg. Previously, a call to
|
|
``request.getfuncargvalue()`` happens at test execution time and
|
|
thus pytest would not know at collection time about the fact that
|
|
a required resource is parametrized.
|
|
|
|
No ``pytest_funcarg__`` prefix when using @funcarg decorator
|
|
-------------------------------------------------------------------
|
|
|
|
|
|
.. note:: Implemented
|
|
|
|
When using the ``@funcarg`` decorator the name of the function
|
|
does not need to (and in fact cannot) use the ``pytest_funcarg__``
|
|
naming::
|
|
|
|
@pytest.mark.funcarg
|
|
def db(request):
|
|
...
|
|
|
|
The name under which the funcarg resource can be requested is ``db``.
|
|
|
|
You can also use the "old" non-decorator way of specifying funcarg factories
|
|
aka::
|
|
|
|
def pytest_funcarg__db(request):
|
|
...
|
|
|
|
It is recommended to use the funcarg-decorator, however.
|
|
|
|
|
|
solving per-session setup / the new @setup marker
|
|
--------------------------------------------------------------
|
|
|
|
.. note:: Implemented, at least working for basic situations.
|
|
|
|
pytest for a long time offered a pytest_configure and a pytest_sessionstart
|
|
hook which are often used to setup global resources. This suffers from
|
|
several problems:
|
|
|
|
1. in distributed testing the master process would setup test resources
|
|
that are never needed because it only co-ordinates the test run
|
|
activities of the slave processes.
|
|
|
|
2. if you only perform a collection (with "--collectonly")
|
|
resource-setup will still be executed.
|
|
|
|
3. If a pytest_sessionstart is contained in some subdirectories
|
|
conftest.py file, it will not be called. This stems from the
|
|
fact that this hook is actually used for reporting, in particular
|
|
the test-header with platform/custom information.
|
|
|
|
Moreover, it is today not easy to define a scoped setup from plugins or
|
|
conftest files other than to implement a ``pytest_runtest_setup()`` hook
|
|
and caring for scoping/caching yourself. And it's virtually impossible
|
|
to do this with parametrization as ``pytest_runtest_setup()`` is called
|
|
during test execution and parametrization happens at collection time.
|
|
|
|
It follows that pytest_configure/session/runtest_setup are often not
|
|
appropriate for implementing common fixture needs. Therefore,
|
|
pytest-2.X introduces a new "@pytest.mark.setup" marker which takes
|
|
an optional "scope" parameter.
|
|
|
|
See :ref:`new_setup` for examples.
|
|
|
|
funcarg and setup discovery now happens at collection time
|
|
---------------------------------------------------------------------
|
|
|
|
.. note::
|
|
Partially implemented - collectonly shows no extra information however.
|
|
|
|
pytest-2.X takes care to discover funcarg factories and @setup methods
|
|
at collection time. This is more efficient especially for large test suites.
|
|
Moreover, a call to "py.test --collectonly" should be able to show
|
|
a lot of setup-information and thus presents a nice method to get an
|
|
overview of resource management in your project.
|
|
|
|
|
|
Grouping tests by resource parameters
|
|
----------------------------------------------------------
|
|
|
|
.. note:: Implemented.
|
|
|
|
pytest used to always sort test items by their source location.
|
|
With pytest-2.X tests are first grouped by funcarg parameters.
|
|
If you have a parametrized funcarg, then all the tests using it
|
|
will first execute with it. Then any finalizers are called and then
|
|
the next parametrized resource instance is created and its tests are run.
|
|
Among other things, this eases testing of applications which create
|
|
and use global state.
|
|
|
|
The following example uses two parametrized funcargs, one of which is
|
|
scoped on a per-module basis::
|
|
|
|
# content of test_module.py
|
|
import pytest
|
|
|
|
@pytest.mark.funcarg(scope="module", params=["mod1", "mod2"])
|
|
def modarg(request):
|
|
param = request.param
|
|
print "create", param
|
|
def fin():
|
|
print "fin", param
|
|
request.addfinalizer(fin)
|
|
return param
|
|
|
|
@pytest.mark.funcarg(scope="function", params=[1,2])
|
|
def otherarg(request):
|
|
return request.param
|
|
|
|
def test_0(otherarg):
|
|
print " test0", otherarg
|
|
def test_1(modarg):
|
|
print " test1", modarg
|
|
def test_2(otherarg, modarg):
|
|
print " test2", otherarg, modarg
|
|
|
|
Let's run the tests in verbose mode and with looking at the print-output::
|
|
|
|
$ py.test -v -s
|
|
=========================== test session starts ============================
|
|
platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev5 -- /home/hpk/venv/1/bin/python
|
|
cachedir: /home/hpk/tmp/doc-exec-388/.cache
|
|
plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov
|
|
collecting ... collected 8 items
|
|
|
|
test_module.py:16: test_0[1] PASSED
|
|
test_module.py:16: test_0[2] PASSED
|
|
test_module.py:18: test_1[mod1] PASSED
|
|
test_module.py:20: test_2[1-mod1] PASSED
|
|
test_module.py:20: test_2[2-mod1] PASSED
|
|
test_module.py:18: test_1[mod2] PASSED
|
|
test_module.py:20: test_2[1-mod2] PASSED
|
|
test_module.py:20: test_2[2-mod2] PASSED
|
|
|
|
========================= 8 passed in 0.03 seconds =========================
|
|
test0 1
|
|
test0 2
|
|
create mod1
|
|
test1 mod1
|
|
test2 1 mod1
|
|
test2 2 mod1
|
|
fin mod1
|
|
create mod2
|
|
test1 mod2
|
|
test2 1 mod2
|
|
test2 2 mod2
|
|
fin mod2
|
|
|
|
You can see that that the parametrized ``modarg`` resource lead to
|
|
a re-ordering of test execution. The finalizer for the "mod1" parametrized
|
|
resource was executed before the "mod2" resource was setup.
|
|
|
|
.. note::
|
|
|
|
The current implementation is experimental.
|