419 lines
14 KiB
Plaintext
419 lines
14 KiB
Plaintext
======================================================
|
|
**funcargs**: test setup and parametrization
|
|
======================================================
|
|
|
|
Since version 1.0 test functions can make great use of
|
|
their arguments or "funcargs" for short. py.test helps
|
|
to setup or generate argument values with the goal
|
|
of making it easy to:
|
|
|
|
* separate test function code from test setup/fixtures
|
|
* manage test value setup and teardown depending on
|
|
command line options or configuration
|
|
* parametrize multiple runs of the same test functions
|
|
* present useful debug info if something goes wrong
|
|
|
|
Using funcargs, test functions become more expressive,
|
|
more "templaty" and more test-aspect oriented. In fact,
|
|
funcarg mechanisms are meant to be complete and
|
|
convenient enough to
|
|
|
|
* substitute most usages of `xUnit style`_ setup
|
|
|
|
* substitute all usages of `old-style generative tests`_,
|
|
i.e. test functions that use the "yield" statement.
|
|
Using yield in test functions is deprecated since 1.0.
|
|
|
|
|
|
.. _`xUnit style`: xunit_setup.html
|
|
.. _`old-style generative tests`:
|
|
|
|
.. _`funcarg provider`:
|
|
|
|
funcarg providers: setting up test function arguments
|
|
==============================================================
|
|
|
|
Test functions can specify one ore more arguments ("funcargs")
|
|
and a test module or plugin can define functions that provide
|
|
the function argument. Let's look at a self-contained example
|
|
that you can put into a test module:
|
|
|
|
.. sourcecode:: python
|
|
|
|
def pytest_funcarg__myfuncarg(request):
|
|
return 42
|
|
|
|
def test_function(myfuncarg):
|
|
assert myfuncarg == 42
|
|
|
|
Here is what happens:
|
|
|
|
1. **lookup funcarg provider**: The ``test_function`` needs an value for
|
|
``myfuncarg`` to run. The provider is found by its special
|
|
name, ``pytest_funcarg__`` followed by the function
|
|
argument argument name. If a provider cannot be found,
|
|
a list of all available function arguments is presented.
|
|
|
|
2. **setup funcarg value**: ``pytest_funcarg__myfuncarg(request)`` is
|
|
called to setup the value for ``myfuncarg``.
|
|
|
|
3. **execute test** ``test_function(42)`` call is executed.
|
|
If the test fails one can see the original provided
|
|
value in the traceback at the top.
|
|
|
|
.. _`request object`:
|
|
|
|
funcarg request objects
|
|
------------------------------------------
|
|
|
|
Request objects are passed to funcarg providers. Request objects
|
|
encapsulate a request for a function argument for a
|
|
specific test function. Request objects allow providers to access
|
|
test configuration and test context:
|
|
|
|
``request.argname``: name of the requested function argument
|
|
|
|
``request.function``: python function object requesting the argument
|
|
|
|
``request.cls``: class object where the test function is defined in or None.
|
|
|
|
``request.module``: module object where the test function is defined in.
|
|
|
|
``request.config``: access to command line opts and general config
|
|
|
|
``request.param``: if exists is the argument passed by a `parametrizing test generator`_
|
|
|
|
|
|
cleanup after test function execution
|
|
---------------------------------------------
|
|
|
|
Request objects allow to **register a finalizer method** which is
|
|
called after a test function has finished running.
|
|
This is useful for tearing down or cleaning up
|
|
test state. Here is a basic example for providing
|
|
a ``myfile`` object that will be closed upon test
|
|
function finish:
|
|
|
|
.. sourcecode:: python
|
|
|
|
def pytest_funcarg__myfile(self, request):
|
|
# ... create and open a "myfile" object ...
|
|
request.addfinalizer(lambda: myfile.close())
|
|
return myfile
|
|
|
|
|
|
decorating other funcarg providers
|
|
++++++++++++++++++++++++++++++++++++++++
|
|
|
|
If you want to **decorate a function argument** that is
|
|
provided elsewhere you can ask the request object
|
|
to provide the "next" value:
|
|
|
|
.. sourcecode:: python
|
|
|
|
def pytest_funcarg__myfile(self, request):
|
|
myfile = request.call_next_provider()
|
|
# do something extra
|
|
return myfile
|
|
|
|
This will raise a ``request.Error`` exception if there
|
|
is no next provider left. See the `decorator example`_
|
|
for a use of this method.
|
|
|
|
|
|
.. _`test generators`:
|
|
.. _`parametrizing test generator`:
|
|
|
|
generating parametrized tests with funcargs
|
|
===========================================================
|
|
|
|
You can parametrize multiple runs of the same test function
|
|
by schedulings new test function calls which get different
|
|
funcarg values. Let's look at a simple self-contained
|
|
example:
|
|
|
|
.. sourcecode:: python
|
|
|
|
def pytest_generate_tests(metafunc):
|
|
if "numiter" in metafunc.funcargs:
|
|
for i in range(10):
|
|
metafunc.addcall(param=i)
|
|
|
|
def pytest_funcarg__numiter(request):
|
|
return request.param
|
|
|
|
def test_func(numiter):
|
|
assert numiter < 10
|
|
|
|
Here is what happens in detail:
|
|
|
|
1. **add test function calls**:
|
|
``pytest_generate_tests(metafunc)`` hook is called once for each test
|
|
function. The `metafunc object`_ has context information.
|
|
``metafunc.addcall(param=i)`` schedules a new test call
|
|
such that function argument providers will see an additional
|
|
``arg`` attribute on their request object.
|
|
|
|
2. **setup funcarg values**: the ``pytest_funcarg__arg1(request)`` provider is called
|
|
10 times with ten different request objects all pointing to
|
|
the same test function. Our provider here simply returns
|
|
the ``arg`` value but we could of course also setup more
|
|
heavyweight resources here.
|
|
|
|
3. **execute tests**: ``test_func(numiter)`` is called ten times with
|
|
ten different arguments.
|
|
|
|
.. _`metafunc object`:
|
|
|
|
test generators and metafunc objects
|
|
-------------------------------------------
|
|
|
|
metafunc objects are passed to the ``pytest_generate_tests`` hook.
|
|
They help to inspect a testfunction and to generate tests
|
|
according to test configuration or values specified
|
|
in the class or module where a test function is defined:
|
|
|
|
``metafunc.funcargnames``: set of required function arguments for given function
|
|
|
|
``metafunc.function``: underlying python test function
|
|
|
|
``metafunc.cls``: class object where the test function is defined in or None.
|
|
|
|
``metafunc.module``: the module object where the test function is defined in.
|
|
|
|
``metafunc.config``: access to command line opts and general config
|
|
|
|
|
|
the ``metafunc.addcall()`` method
|
|
-----------------------------------------------
|
|
|
|
.. sourcecode:: python
|
|
|
|
def addcall(id=None, param=None):
|
|
""" trigger a later test function call. """
|
|
|
|
The specified ``param`` will be seen by the
|
|
`funcarg provider`_ as a ``request.param`` attribute.
|
|
|
|
If you provide an `id`` it will be used for reporting
|
|
and identification purposes. If you don't supply an `id`
|
|
the stringified counter of the list of added calls will be used.
|
|
``id`` values needs to be unique between all
|
|
invocations for a given test function.
|
|
|
|
*Test generators are called during test collection which
|
|
is separate from the actual test setup and test run.
|
|
With distributed testing setting up funcargs will
|
|
even happen in a different process. Therefore one should
|
|
defer setup of heavyweight objects to funcarg providers.*
|
|
|
|
|
|
Funcarg Tutorial Examples
|
|
=======================================
|
|
|
|
application specific test setup
|
|
---------------------------------------------------------
|
|
|
|
Here is a basic useful step-wise example for handling application
|
|
specific test setup. The goal is to have one place where we have the
|
|
glue code for bootstrapping and configuring application objects and allow
|
|
test modules and test functions to stay ignorant of involved details.
|
|
|
|
step 1: use and implement a test/app-specific "mysetup"
|
|
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
|
|
|
Let's write a simple test function living in a test file
|
|
``test_sample.py`` that uses a ``mysetup`` funcarg for accessing test
|
|
specific setup.
|
|
|
|
.. sourcecode:: python
|
|
|
|
# ./test_sample.py
|
|
def test_answer(mysetup):
|
|
app = mysetup.myapp()
|
|
answer = app.question()
|
|
assert answer == 42
|
|
|
|
To run this test py.test needs to find and call a provider to
|
|
obtain the required ``mysetup`` function argument. The test
|
|
function interacts with the provided application specific setup.
|
|
|
|
To provide the ``mysetup`` function argument we write down
|
|
a provider method in a `local plugin`_ by putting the
|
|
following code into a local ``conftest.py``:
|
|
|
|
.. sourcecode:: python
|
|
|
|
# ./conftest.py
|
|
from myapp import MyApp
|
|
|
|
class ConftestPlugin:
|
|
def pytest_funcarg__mysetup(self, request):
|
|
return MySetup()
|
|
|
|
class MySetup:
|
|
def myapp(self):
|
|
return MyApp()
|
|
|
|
To run the example we put a pseudo MyApp object into ``myapp.py``:
|
|
|
|
.. sourcecode:: python
|
|
|
|
# ./myapp.py
|
|
class MyApp:
|
|
def question(self):
|
|
return 6 * 9
|
|
|
|
You can now run the test with ``py.test test_sample.py`` which will
|
|
show this failure:
|
|
|
|
.. sourcecode:: python
|
|
|
|
def test_answer(mysetup):
|
|
app = mysetup.myapp()
|
|
answer = app.question()
|
|
> assert answer == 42
|
|
E assert 54 == 42
|
|
|
|
If you are confused as to what the concrete question or answers
|
|
mean actually, please visit here_ :)
|
|
|
|
.. _here: http://uncyclopedia.wikia.com/wiki/The_Hitchhiker's_Guide_to_the_Galaxy
|
|
.. _`local plugin`: ext.html#local-plugin
|
|
|
|
|
|
step 2: adding a command line option
|
|
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
|
|
|
If you provide a "funcarg" from a plugin you can easily make methods
|
|
depend on command line options or environment settings. Let's write a
|
|
local plugin that adds a command line option to ``py.test`` invocations:
|
|
|
|
.. sourcecode:: python
|
|
|
|
class ConftestPlugin:
|
|
def pytest_addoption(self, parser):
|
|
parser.addoption("--ssh", action="store", default=None,
|
|
help="specify ssh host to run tests with")
|
|
|
|
pytest_funcarg__mysetup = MySetupFuncarg
|
|
|
|
class MySetupFuncarg:
|
|
def __init__(self, request):
|
|
self.request = request
|
|
def getsshconnection(self):
|
|
host = self.request.config.option.ssh
|
|
if host is None:
|
|
py.test.skip("specify ssh host with --ssh to run this test")
|
|
return py.execnet.SshGateway(host)
|
|
|
|
Now any test functions can use the ``mysetup.getsshconnection()`` method like this:
|
|
|
|
.. sourcecode:: python
|
|
|
|
class TestClass:
|
|
def test_function(self, mysetup):
|
|
conn = mysetup.getsshconnection()
|
|
# work with conn
|
|
|
|
Running this without specifying a command line option will result in a skipped
|
|
test_function.
|
|
|
|
.. _`accept example`:
|
|
|
|
example: specifying and selecting acceptance tests
|
|
--------------------------------------------------------------
|
|
|
|
.. sourcecode:: python
|
|
|
|
class ConftestPlugin:
|
|
def pytest_option(self, parser):
|
|
group = parser.getgroup("myproject")
|
|
group.addoption("-A", dest="acceptance", action="store_true",
|
|
help="run (slow) acceptance tests")
|
|
|
|
def pytest_funcarg__accept(self, request):
|
|
return AcceptFuncarg(request)
|
|
|
|
class AcceptFuncarg:
|
|
def __init__(self, request):
|
|
if not request.config.option.acceptance:
|
|
py.test.skip("specify -A to run acceptance tests")
|
|
self.tmpdir = request.config.mktemp(request.function.__name__, numbered=True)
|
|
|
|
def run(self, cmd):
|
|
""" called by test code to execute an acceptance test. """
|
|
self.tmpdir.chdir()
|
|
return py.process.cmdexec(cmd)
|
|
|
|
|
|
and the actual test function example:
|
|
|
|
.. sourcecode:: python
|
|
|
|
def test_some_acceptance_aspect(accept):
|
|
accept.tmpdir.mkdir("somesub")
|
|
result = accept.run("ls -la")
|
|
assert "somesub" in result
|
|
|
|
If you run this test without specifying a command line option
|
|
the test will get skipped with an appropriate message. Otherwise
|
|
you can start to add convenience and test support methods
|
|
to your AcceptFuncarg and drive running of tools or
|
|
applications and provide ways to do assertions about
|
|
the output.
|
|
|
|
.. _`decorator example`:
|
|
|
|
example: decorating a funcarg in a test module
|
|
--------------------------------------------------------------
|
|
|
|
For larger scale setups it's sometimes useful to decorare
|
|
a funcarg just for a particular test module. We can
|
|
extend the `accept example`_ by putting this in our test class:
|
|
|
|
.. sourcecode:: python
|
|
|
|
def pytest_funcarg__accept(self, request):
|
|
arg = request.call_next_provider()
|
|
# create a special layout in our tempdir
|
|
arg.tmpdir.mkdir("special")
|
|
return arg
|
|
|
|
class TestSpecialAcceptance:
|
|
def test_sometest(self, accept):
|
|
assert accept.tmpdir.join("special").check()
|
|
|
|
Our module level provider will be invoked first and it can
|
|
ask its request object to call the next provider and then
|
|
decorate its result. This mechanism allows us to stay
|
|
ignorant of how/where the function argument is provided -
|
|
in our example from a ConftestPlugin but could be any plugin.
|
|
|
|
sidenote: the temporary directory used here are instances of
|
|
the `py.path.local`_ class which provides many of the os.path
|
|
methods in a convenient way.
|
|
|
|
.. _`py.path.local`: ../path.html#local
|
|
|
|
|
|
Questions and Answers
|
|
==================================
|
|
|
|
.. _`why pytest_pyfuncarg__ methods?`:
|
|
|
|
Why ``pytest_funcarg__*`` methods?
|
|
------------------------------------
|
|
|
|
When experimenting with funcargs we also
|
|
considered an explicit registration mechanism, i.e. calling a register
|
|
method on the config object. But lacking a good use case for this
|
|
indirection and flexibility we decided to go for `Convention over
|
|
Configuration`_ and allow to directly specify the provider. It has the
|
|
positive implication that you should be able to "grep" for
|
|
``pytest_funcarg__MYARG`` and will find all providing sites (usually
|
|
exactly one).
|
|
|
|
.. _`Convention over Configuration`: http://en.wikipedia.org/wiki/Convention_over_Configuration
|
|
|