309 lines
11 KiB
Plaintext
309 lines
11 KiB
Plaintext
|
|
The writing and reporting of assertions in tests
|
|
==================================================
|
|
|
|
.. _`assert with the assert statement`:
|
|
|
|
assert with the ``assert`` statement
|
|
---------------------------------------------------------
|
|
|
|
``py.test`` allows you to use the standard python ``assert`` for verifying
|
|
expectations and values in Python tests. For example, you can write the
|
|
following::
|
|
|
|
# content of test_assert1.py
|
|
def f():
|
|
return 3
|
|
|
|
def test_function():
|
|
assert f() == 4
|
|
|
|
to assert that your function returns a certain value. If this assertion fails
|
|
you will see the return value of the function call::
|
|
|
|
$ py.test test_assert1.py
|
|
=========================== test session starts ============================
|
|
platform linux2 -- Python 2.6.6 -- pytest-2.0.3
|
|
collecting ... collected 1 items
|
|
|
|
test_assert1.py F
|
|
|
|
================================= FAILURES =================================
|
|
______________________________ test_function _______________________________
|
|
|
|
def test_function():
|
|
> assert f() == 4
|
|
E assert 3 == 4
|
|
E + where 3 = f()
|
|
|
|
test_assert1.py:5: AssertionError
|
|
========================= 1 failed in 0.02 seconds =========================
|
|
|
|
py.test has support for showing the values of the most common subexpressions
|
|
including calls, attributes, comparisons, and binary and unary
|
|
operators. (See :ref:`tbreportdemo`). This allows you to use the
|
|
idiomatic python constructs without boilerplate code while not losing
|
|
introspection information.
|
|
|
|
However, if you specify a message with the assertion like this::
|
|
|
|
assert a % 2 == 0, "value was odd, should be even"
|
|
|
|
then no assertion introspection takes places at all and the message
|
|
will be simply shown in the traceback.
|
|
|
|
See :ref:`assert-details` for more information on assertion introspection.
|
|
|
|
assertions about expected exceptions
|
|
------------------------------------------
|
|
|
|
In order to write assertions about raised exceptions, you can use
|
|
``pytest.raises`` as a context manager like this::
|
|
|
|
import pytest
|
|
with pytest.raises(ZeroDivisionError):
|
|
1 / 0
|
|
|
|
and if you need to have access to the actual exception info you may use::
|
|
|
|
with pytest.raises(RuntimeError) as excinfo:
|
|
def f():
|
|
f()
|
|
f()
|
|
|
|
# do checks related to excinfo.type, excinfo.value, excinfo.traceback
|
|
|
|
If you want to write test code that works on Python2.4 as well,
|
|
you may also use two other ways to test for an expected exception::
|
|
|
|
pytest.raises(ExpectedException, func, *args, **kwargs)
|
|
pytest.raises(ExpectedException, "func(*args, **kwargs)")
|
|
|
|
both of which execute the specified function with args and kwargs and
|
|
asserts that the given ``ExpectedException`` is raised. The reporter will
|
|
provide you with helpful output in case of failures such as *no
|
|
exception* or *wrong exception*.
|
|
|
|
.. _newreport:
|
|
|
|
Making use of context-sensitive comparisons
|
|
-------------------------------------------------
|
|
|
|
.. versionadded:: 2.0
|
|
|
|
py.test has rich support for providing context-sensitive information
|
|
when it encounters comparisons. For example::
|
|
|
|
# content of test_assert2.py
|
|
|
|
def test_set_comparison():
|
|
set1 = set("1308")
|
|
set2 = set("8035")
|
|
assert set1 == set2
|
|
|
|
if you run this module::
|
|
|
|
$ py.test test_assert2.py
|
|
=========================== test session starts ============================
|
|
platform linux2 -- Python 2.6.6 -- pytest-2.0.3
|
|
collecting ... collected 1 items
|
|
|
|
test_assert2.py F
|
|
|
|
================================= FAILURES =================================
|
|
___________________________ test_set_comparison ____________________________
|
|
|
|
def test_set_comparison():
|
|
set1 = set("1308")
|
|
set2 = set("8035")
|
|
> assert set1 == set2
|
|
E assert set(['0', '1', '3', '8']) == set(['0', '3', '5', '8'])
|
|
E Extra items in the left set:
|
|
E '1'
|
|
E Extra items in the right set:
|
|
E '5'
|
|
|
|
test_assert2.py:5: AssertionError
|
|
========================= 1 failed in 0.03 seconds =========================
|
|
|
|
Special comparisons are done for a number of cases:
|
|
|
|
* comparing long strings: a context diff is shown
|
|
* comparing long sequences: first failing indices
|
|
* comparing dicts: different entries
|
|
|
|
See the :ref:`reporting demo <tbreportdemo>` for many more examples.
|
|
|
|
|
|
.. _assert-details:
|
|
.. _`assert introspection`:
|
|
|
|
Advanced assertion introspection
|
|
----------------------------------
|
|
|
|
.. versionadded:: 2.1
|
|
|
|
|
|
Reporting details about a failing assertion is achieved either by rewriting
|
|
assert statements before they are run or re-evaluating the assert expression and
|
|
recording the intermediate values. Which technique is used depends on the
|
|
location of the assert, py.test's configuration, and Python version being used
|
|
to run py.test. However, for assert statements with a manually provided
|
|
message, i.e. ``assert expr, message``, no assertion introspection takes place and the manually provided message will be rendered in tracebacks.
|
|
|
|
By default, if the Python version is greater than or equal to 2.6, py.test
|
|
rewrites assert statements in test modules. Rewritten assert statements put
|
|
introspection information into the assertion failure message. py.test only
|
|
rewrites test modules directly discovered by its test collection process, so
|
|
asserts in supporting modules will not be rewritten.
|
|
|
|
.. note::
|
|
|
|
py.test rewrites test modules as it collects tests from them. It does this by
|
|
writing a new pyc file which Python loads when the test module is
|
|
imported. If the module has already been loaded (it is in sys.modules),
|
|
though, Python will not load the rewritten module. This means if a test
|
|
module imports another test module which has not already been rewritten, then
|
|
py.test will not be able to rewrite the second module.
|
|
|
|
If an assert statement has not been rewritten or the Python version is less than
|
|
2.6, py.test falls back on assert reinterpretation. In assert reinterpretation,
|
|
py.test walks the frame of the function containing the assert statement to
|
|
discover sub-expression results of the failing assert statement. You can force
|
|
py.test to always use assertion reinterpretation by passing the
|
|
``--assertmode=old`` option.
|
|
|
|
Assert reinterpretation has a caveat not present with assert rewriting: If
|
|
evaluating the assert expression has side effects you may get a warning that the
|
|
intermediate values could not be determined safely. A common example of this
|
|
issue is an assertion which reads from a file::
|
|
|
|
assert f.read() != '...'
|
|
|
|
If this assertion fails then the re-evaluation will probably succeed!
|
|
This is because ``f.read()`` will return an empty string when it is
|
|
called the second time during the re-evaluation. However, it is
|
|
easy to rewrite the assertion and avoid any trouble::
|
|
|
|
content = f.read()
|
|
assert content != '...'
|
|
|
|
All assert introspection can be turned off by passing ``--assertmode=off``.
|
|
|
|
.. versionadded:: 2.1
|
|
Add assert rewriting as an alternate introspection technique.
|
|
|
|
.. versionchanged:: 2.1
|
|
Introduce the ``--assertmode`` option. Deprecate ``--no-assert`` and
|
|
``--nomagic``.
|
|
|
|
|
|
Defining your own comparison
|
|
----------------------------------------------
|
|
|
|
|
|
As already shown in examples py.test is able to provide detailed
|
|
explanations when an assertial fails. E.g. when comparing a
|
|
dictionary it will show you which elements differ::
|
|
|
|
$ py.test example.py
|
|
============================= test session starts ==========================
|
|
platform linux2 -- Python 2.7.1 -- pytest-2.0.3
|
|
collected 1 items
|
|
|
|
example.py F
|
|
|
|
=================================== FAILURES ===============================
|
|
__________________________________ test_text _______________________________
|
|
|
|
def test_text():
|
|
> assert {'foo': 0, 'bar': 1} == {'foo': 0, 'bar': 0}
|
|
E assert {'bar': 1, 'foo': 0} == {'bar': 0, 'foo': 0}
|
|
E - {'bar': 1, 'foo': 0}
|
|
E ? ^
|
|
E + {'bar': 0, 'foo': 0}
|
|
E ? ^
|
|
|
|
example.py:2: AssertionError
|
|
=========================== 1 failed in 0.03 seconds =======================
|
|
|
|
|
|
py.test has builtin knowledge about displaying detailed information
|
|
for a number of types. If the objects compared do not match those it
|
|
will fall back to a less detailed genric comparison, e.g.::
|
|
|
|
$ py.test example2.py
|
|
============================= test session starts ==========================
|
|
platform linux2 -- Python 2.7.1 -- pytest-2.0.3
|
|
collected 1 items
|
|
|
|
example2.py F
|
|
|
|
=================================== FAILURES ===============================
|
|
________________________________ test_foo __________________________________
|
|
|
|
def test_foo():
|
|
f = Foo(1)
|
|
g = Foo(2)
|
|
> assert f == g
|
|
E assert <example2.Foo object at 0x190de90> == <example2.Foo object at 0x190df10>
|
|
|
|
example2.py:24: AssertionError
|
|
=========================== 1 failed in 0.03 seconds =======================
|
|
|
|
|
|
The detailed builtin comparisons are currently only present for
|
|
strings, sequences, sets and dictionaries which are compared for
|
|
equality (``==``) and for strings with a ``not in`` comparison.
|
|
However it is possible to add your own detailed explanations using the
|
|
``pytest_assertrepr_compare`` hook.
|
|
|
|
.. py:function:: pytest_assertrepr_compare(config, op, left, right)
|
|
|
|
The *config* argument is a ``_pytest.config.Config``
|
|
instance. *op* will be the comparison operator: ``==``, ``<``,
|
|
``in``, etc. While *left* and *right* will contain the objects
|
|
which are being compared.
|
|
|
|
The return value must be either *None* in case this hook does not
|
|
provide an explanation for the arguments passed in, or a list of
|
|
strings. Each string in the list will be regarded as a line and
|
|
displayed on a line by itself. The first line is slightly special
|
|
and is meant to be a summary with the detailed explanation
|
|
following on the other lines.
|
|
|
|
|
|
As an example consider adding following hook in a conftest.py which
|
|
provides an alternative explanation for the ``Foo`` type used above::
|
|
|
|
def pytest_assertrepr_compare(op, left, right):
|
|
if (not isinstance(left, example2.Foo) or
|
|
not isinstance(right, example2.Foo) or
|
|
op != '=='):
|
|
return None
|
|
return ['Comparing Foo instances:',
|
|
' vals: %s != %s' % (left.val, right.val)]
|
|
|
|
When re-running the above example after adding this in the conftest.py
|
|
file the output now contains a more detailed description::
|
|
|
|
$ python pytest.py example2.py
|
|
============================= test session starts ==========================
|
|
platform linux2 -- Python 2.7.1 -- pytest-2.1.0.dev4
|
|
collected 1 items
|
|
|
|
example2.py F
|
|
|
|
=================================== FAILURES ===============================
|
|
___________________________________ test_foo _______________________________
|
|
|
|
def test_foo():
|
|
f = Foo(1)
|
|
g = Foo(2)
|
|
> assert f == g
|
|
E assert Comparing Foo instances:
|
|
E vals: 1 != 2
|
|
|
|
example2.py:24: AssertionError
|
|
=========================== 1 failed in 0.03 seconds =======================
|