e368fb4b29
this allows a clear addition of parameterization parameters that carry along marks instead of nesting multiple mark objects and destroying the possibility of creating function valued parameters, it just folders everything together into one object carrfying parameters, and the marks.
240 lines
7.6 KiB
ReStructuredText
240 lines
7.6 KiB
ReStructuredText
|
|
.. _`test generators`:
|
|
.. _`parametrizing-tests`:
|
|
.. _`parametrized test functions`:
|
|
.. _`parametrize`:
|
|
|
|
.. _`parametrize-basics`:
|
|
|
|
Parametrizing fixtures and test functions
|
|
==========================================================================
|
|
|
|
pytest supports test parametrization in several well-integrated ways:
|
|
|
|
- :py:func:`pytest.fixture` allows to define :ref:`parametrization
|
|
at the level of fixture functions <fixture-parametrize>`.
|
|
|
|
* `@pytest.mark.parametrize`_ allows to define parametrization at the
|
|
function or class level, provides multiple argument/fixture sets
|
|
for a particular test function or class.
|
|
|
|
* `pytest_generate_tests`_ enables implementing your own custom
|
|
dynamic parametrization scheme or extensions.
|
|
|
|
.. _parametrizemark:
|
|
.. _`@pytest.mark.parametrize`:
|
|
|
|
|
|
``@pytest.mark.parametrize``: parametrizing test functions
|
|
---------------------------------------------------------------------
|
|
|
|
.. regendoc: wipe
|
|
|
|
.. versionadded:: 2.2
|
|
.. versionchanged:: 2.4
|
|
Several improvements.
|
|
|
|
The builtin ``pytest.mark.parametrize`` decorator enables
|
|
parametrization of arguments for a test function. Here is a typical example
|
|
of a test function that implements checking that a certain input leads
|
|
to an expected output::
|
|
|
|
# content of test_expectation.py
|
|
import pytest
|
|
@pytest.mark.parametrize("test_input,expected", [
|
|
("3+5", 8),
|
|
("2+4", 6),
|
|
("6*9", 42),
|
|
])
|
|
def test_eval(test_input, expected):
|
|
assert eval(test_input) == expected
|
|
|
|
Here, the ``@parametrize`` decorator defines three different ``(test_input,expected)``
|
|
tuples so that the ``test_eval`` function will run three times using
|
|
them in turn::
|
|
|
|
$ pytest
|
|
======= test session starts ========
|
|
platform linux -- Python 3.5.2, pytest-3.0.3, py-1.4.31, pluggy-0.4.0
|
|
rootdir: $REGENDOC_TMPDIR, inifile:
|
|
collected 3 items
|
|
|
|
test_expectation.py ..F
|
|
|
|
======= FAILURES ========
|
|
_______ test_eval[6*9-42] ________
|
|
|
|
test_input = '6*9', expected = 42
|
|
|
|
@pytest.mark.parametrize("test_input,expected", [
|
|
("3+5", 8),
|
|
("2+4", 6),
|
|
("6*9", 42),
|
|
])
|
|
def test_eval(test_input, expected):
|
|
> assert eval(test_input) == expected
|
|
E assert 54 == 42
|
|
E + where 54 = eval('6*9')
|
|
|
|
test_expectation.py:8: AssertionError
|
|
======= 1 failed, 2 passed in 0.12 seconds ========
|
|
|
|
As designed in this example, only one pair of input/output values fails
|
|
the simple test function. And as usual with test function arguments,
|
|
you can see the ``input`` and ``output`` values in the traceback.
|
|
|
|
Note that you could also use the parametrize marker on a class or a module
|
|
(see :ref:`mark`) which would invoke several functions with the argument sets.
|
|
|
|
It is also possible to mark individual test instances within parametrize,
|
|
for example with the builtin ``mark.xfail``::
|
|
|
|
# content of test_expectation.py
|
|
import pytest
|
|
@pytest.mark.parametrize("test_input,expected", [
|
|
("3+5", 8),
|
|
("2+4", 6),
|
|
pytest.param("6*9", 42,
|
|
marks=pytest.mark.xfail),
|
|
])
|
|
def test_eval(test_input, expected):
|
|
assert eval(test_input) == expected
|
|
|
|
.. note::
|
|
|
|
prior to version 3.1 the supported mechanism for marking values
|
|
used the syntax::
|
|
|
|
import pytest
|
|
@pytest.mark.parametrize("test_input,expected", [
|
|
("3+5", 8),
|
|
("2+4", 6),
|
|
pytest.mark.xfail(("6*9", 42),),
|
|
])
|
|
def test_eval(test_input, expected):
|
|
assert eval(test_input) == expected
|
|
|
|
|
|
This was an initial hack to support the feature but soon was demonstrated to be incomplete,
|
|
broken for passing functions or applying multiple marks with the same name but different parameters.
|
|
The old syntax will be removed in pytest-4.0.
|
|
|
|
|
|
Let's run this::
|
|
|
|
$ pytest
|
|
======= test session starts ========
|
|
platform linux -- Python 3.5.2, pytest-3.0.3, py-1.4.31, pluggy-0.4.0
|
|
rootdir: $REGENDOC_TMPDIR, inifile:
|
|
collected 3 items
|
|
|
|
test_expectation.py ..x
|
|
|
|
======= 2 passed, 1 xfailed in 0.12 seconds ========
|
|
|
|
The one parameter set which caused a failure previously now
|
|
shows up as an "xfailed (expected to fail)" test.
|
|
|
|
To get all combinations of multiple parametrized arguments you can stack
|
|
``parametrize`` decorators::
|
|
|
|
import pytest
|
|
@pytest.mark.parametrize("x", [0, 1])
|
|
@pytest.mark.parametrize("y", [2, 3])
|
|
def test_foo(x, y):
|
|
pass
|
|
|
|
This will run the test with the arguments set to x=0/y=2, x=0/y=3, x=1/y=2 and
|
|
x=1/y=3.
|
|
|
|
.. note::
|
|
|
|
In versions prior to 2.4 one needed to specify the argument
|
|
names as a tuple. This remains valid but the simpler ``"name1,name2,..."``
|
|
comma-separated-string syntax is now advertised first because
|
|
it's easier to write and produces less line noise.
|
|
|
|
.. _`pytest_generate_tests`:
|
|
|
|
Basic ``pytest_generate_tests`` example
|
|
---------------------------------------------
|
|
|
|
Sometimes you may want to implement your own parametrization scheme
|
|
or implement some dynamism for determining the parameters or scope
|
|
of a fixture. For this, you can use the ``pytest_generate_tests`` hook
|
|
which is called when collecting a test function. Through the passed in
|
|
``metafunc`` object you can inspect the requesting test context and, most
|
|
importantly, you can call ``metafunc.parametrize()`` to cause
|
|
parametrization.
|
|
|
|
For example, let's say we want to run a test taking string inputs which
|
|
we want to set via a new ``pytest`` command line option. Let's first write
|
|
a simple test accepting a ``stringinput`` fixture function argument::
|
|
|
|
# content of test_strings.py
|
|
|
|
def test_valid_string(stringinput):
|
|
assert stringinput.isalpha()
|
|
|
|
Now we add a ``conftest.py`` file containing the addition of a
|
|
command line option and the parametrization of our test function::
|
|
|
|
# content of conftest.py
|
|
|
|
def pytest_addoption(parser):
|
|
parser.addoption("--stringinput", action="append", default=[],
|
|
help="list of stringinputs to pass to test functions")
|
|
|
|
def pytest_generate_tests(metafunc):
|
|
if 'stringinput' in metafunc.fixturenames:
|
|
metafunc.parametrize("stringinput",
|
|
metafunc.config.option.stringinput)
|
|
|
|
If we now pass two stringinput values, our test will run twice::
|
|
|
|
$ pytest -q --stringinput="hello" --stringinput="world" test_strings.py
|
|
..
|
|
2 passed in 0.12 seconds
|
|
|
|
Let's also run with a stringinput that will lead to a failing test::
|
|
|
|
$ pytest -q --stringinput="!" test_strings.py
|
|
F
|
|
======= FAILURES ========
|
|
_______ test_valid_string[!] ________
|
|
|
|
stringinput = '!'
|
|
|
|
def test_valid_string(stringinput):
|
|
> assert stringinput.isalpha()
|
|
E assert False
|
|
E + where False = <built-in method isalpha of str object at 0xdeadbeef>()
|
|
E + where <built-in method isalpha of str object at 0xdeadbeef> = '!'.isalpha
|
|
|
|
test_strings.py:3: AssertionError
|
|
1 failed in 0.12 seconds
|
|
|
|
As expected our test function fails.
|
|
|
|
If you don't specify a stringinput it will be skipped because
|
|
``metafunc.parametrize()`` will be called with an empty parameter
|
|
list::
|
|
|
|
$ pytest -q -rs test_strings.py
|
|
s
|
|
======= short test summary info ========
|
|
SKIP [1] test_strings.py:1: got empty parameter set ['stringinput'], function test_valid_string at $REGENDOC_TMPDIR/test_strings.py:1
|
|
1 skipped in 0.12 seconds
|
|
|
|
For further examples, you might want to look at :ref:`more
|
|
parametrization examples <paramexamples>`.
|
|
|
|
.. _`metafunc object`:
|
|
|
|
The **metafunc** object
|
|
-------------------------------------------
|
|
|
|
.. currentmodule:: _pytest.python
|
|
.. autoclass:: Metafunc
|
|
:members:
|