473 lines
15 KiB
Plaintext
473 lines
15 KiB
Plaintext
.. _plugins:
|
|
|
|
Working with plugins and conftest files
|
|
=============================================
|
|
|
|
``pytest`` implements all aspects of configuration, collection, running and reporting by calling `well specified hooks`_. Virtually any Python module can be registered as a plugin. It can implement any number of hook functions (usually two or three) which all have a ``pytest_`` prefix, making hook functions easy to distinguish and find. There are three basic location types:
|
|
|
|
* `builtin plugins`_: loaded from pytest's internal ``_pytest`` directory.
|
|
* `external plugins`_: modules discovered through `setuptools entry points`_
|
|
* `conftest.py plugins`_: modules auto-discovered in test directories
|
|
|
|
.. _`pytest/plugin`: http://bitbucket.org/hpk42/pytest/src/tip/pytest/plugin/
|
|
.. _`conftest.py plugins`:
|
|
.. _`conftest.py`:
|
|
.. _`localplugin`:
|
|
.. _`conftest`:
|
|
|
|
conftest.py: local per-directory plugins
|
|
--------------------------------------------------------------
|
|
|
|
local ``conftest.py`` plugins contain directory-specific hook
|
|
implementations. Session and test running activities will
|
|
invoke all hooks defined in ``conftest.py`` files closer to the
|
|
root of the filesystem. Example: Assume the following layout
|
|
and content of files::
|
|
|
|
a/conftest.py:
|
|
def pytest_runtest_setup(item):
|
|
# called for running each test in 'a' directory
|
|
print ("setting up", item)
|
|
|
|
a/test_sub.py:
|
|
def test_sub():
|
|
pass
|
|
|
|
test_flat.py:
|
|
def test_flat():
|
|
pass
|
|
|
|
Here is how you might run it::
|
|
|
|
py.test test_flat.py # will not show "setting up"
|
|
py.test a/test_sub.py # will show "setting up"
|
|
|
|
.. Note::
|
|
If you have ``conftest.py`` files which do not reside in a
|
|
python package directory (i.e. one containing an ``__init__.py``) then
|
|
"import conftest" can be ambiguous because there might be other
|
|
``conftest.py`` files as well on your PYTHONPATH or ``sys.path``.
|
|
It is thus good practise for projects to either put ``conftest.py``
|
|
under a package scope or to never import anything from a
|
|
conftest.py file.
|
|
|
|
.. _`external plugins`:
|
|
.. _`extplugins`:
|
|
|
|
Installing External Plugins / Searching
|
|
------------------------------------------------------
|
|
|
|
Installing a plugin happens through any usual Python installation
|
|
tool, for example::
|
|
|
|
pip install pytest-NAME
|
|
pip uninstall pytest-NAME
|
|
|
|
If a plugin is installed, ``pytest`` automatically finds and integrates it,
|
|
there is no need to activate it. We have a :doc:`page listing
|
|
all 3rd party plugins and their status against the latest py.test version
|
|
<plugins_index/index>` and here is a little annotated list
|
|
for some popular plugins:
|
|
|
|
.. _`django`: https://www.djangoproject.com/
|
|
|
|
* `pytest-django <http://pypi.python.org/pypi/pytest-django>`_: write tests
|
|
for `django`_ apps, using pytest integration.
|
|
|
|
* `pytest-twisted <http://pypi.python.org/pypi/pytest-twisted>`_: write tests
|
|
for `twisted <http://twistedmatrix.com>`_ apps, starting a reactor and
|
|
processing deferreds from test functions.
|
|
|
|
* `pytest-capturelog <http://pypi.python.org/pypi/pytest-capturelog>`_:
|
|
to capture and assert about messages from the logging module
|
|
|
|
* `pytest-cov <http://pypi.python.org/pypi/pytest-cov>`_:
|
|
coverage reporting, compatible with distributed testing
|
|
|
|
* `pytest-xdist <http://pypi.python.org/pypi/pytest-xdist>`_:
|
|
to distribute tests to CPUs and remote hosts, to run in boxed
|
|
mode which allows to survive segmentation faults, to run in
|
|
looponfailing mode, automatically re-running failing tests
|
|
on file changes, see also :ref:`xdist`
|
|
|
|
* `pytest-instafail <http://pypi.python.org/pypi/pytest-instafail>`_:
|
|
to report failures while the test run is happening.
|
|
|
|
* `pytest-bdd <http://pypi.python.org/pypi/pytest-bdd>`_ and
|
|
`pytest-konira <http://pypi.python.org/pypi/pytest-konira>`_
|
|
to write tests using behaviour-driven testing.
|
|
|
|
* `pytest-timeout <http://pypi.python.org/pypi/pytest-timeout>`_:
|
|
to timeout tests based on function marks or global definitions.
|
|
|
|
* `pytest-cache <http://pypi.python.org/pypi/pytest-cache>`_:
|
|
to interactively re-run failing tests and help other plugins to
|
|
store test run information across invocations.
|
|
|
|
* `pytest-pep8 <http://pypi.python.org/pypi/pytest-pep8>`_:
|
|
a ``--pep8`` option to enable PEP8 compliance checking.
|
|
|
|
* `oejskit <http://pypi.python.org/pypi/oejskit>`_:
|
|
a plugin to run javascript unittests in life browsers
|
|
|
|
To see a complete list of all plugins with their latest testing
|
|
status against different py.test and Python versions, please visit
|
|
`pytest-plugs <http://pytest-plugs.herokuapp.com/>`_.
|
|
|
|
You may also discover more plugins through a `pytest- pypi.python.org search`_.
|
|
|
|
.. _`available installable plugins`:
|
|
.. _`pytest- pypi.python.org search`: http://pypi.python.org/pypi?%3Aaction=search&term=pytest-&submit=search
|
|
|
|
Writing a plugin by looking at examples
|
|
------------------------------------------------------
|
|
|
|
.. _`Distribute`: http://pypi.python.org/pypi/distribute
|
|
.. _`setuptools`: http://pypi.python.org/pypi/setuptools
|
|
|
|
If you want to write a plugin, there are many real-life examples
|
|
you can copy from:
|
|
|
|
* a custom collection example plugin: :ref:`yaml plugin`
|
|
* around 20 `builtin plugins`_ which provide pytest's own functionality
|
|
* many `external plugins`_ providing additional features
|
|
|
|
All of these plugins implement the documented `well specified hooks`_
|
|
to extend and add functionality.
|
|
|
|
.. _`setuptools entry points`:
|
|
|
|
Making your plugin installable by others
|
|
-----------------------------------------------
|
|
|
|
If you want to make your plugin externally available, you
|
|
may define a so-called entry point for your distribution so
|
|
that ``pytest`` finds your plugin module. Entry points are
|
|
a feature that is provided by `setuptools`_ or `Distribute`_.
|
|
pytest looks up the ``pytest11`` entrypoint to discover its
|
|
plugins and you can thus make your plugin available by defining
|
|
it in your setuptools/distribute-based setup-invocation:
|
|
|
|
.. sourcecode:: python
|
|
|
|
# sample ./setup.py file
|
|
from setuptools import setup
|
|
|
|
setup(
|
|
name="myproject",
|
|
packages = ['myproject']
|
|
|
|
# the following makes a plugin available to pytest
|
|
entry_points = {
|
|
'pytest11': [
|
|
'name_of_plugin = myproject.pluginmodule',
|
|
]
|
|
},
|
|
)
|
|
|
|
If a package is installed this way, ``pytest`` will load
|
|
``myproject.pluginmodule`` as a plugin which can define
|
|
`well specified hooks`_.
|
|
|
|
.. _`pluginorder`:
|
|
|
|
Plugin discovery order at tool startup
|
|
--------------------------------------------
|
|
|
|
``pytest`` loads plugin modules at tool startup in the following way:
|
|
|
|
* by loading all builtin plugins
|
|
|
|
* by loading all plugins registered through `setuptools entry points`_.
|
|
|
|
* by pre-scanning the command line for the ``-p name`` option
|
|
and loading the specified plugin before actual command line parsing.
|
|
|
|
* by loading all :file:`conftest.py` files as inferred by the command line
|
|
invocation:
|
|
|
|
- if no test paths are specified use current dir as a test path
|
|
- if exists, load ``conftest.py`` and ``test*/conftest.py`` relative
|
|
to the directory part of the first test path.
|
|
|
|
Note that pytest does not find ``conftest.py`` files in deeper nested
|
|
sub directories at tool startup. It is usually a good idea to keep
|
|
your conftest.py file in the top level test or project root directory.
|
|
|
|
* by recursively loading all plugins specified by the
|
|
``pytest_plugins`` variable in ``conftest.py`` files
|
|
|
|
|
|
Requiring/Loading plugins in a test module or conftest file
|
|
-------------------------------------------------------------
|
|
|
|
You can require plugins in a test module or a conftest file like this::
|
|
|
|
pytest_plugins = "name1", "name2",
|
|
|
|
When the test module or conftest plugin is loaded the specified plugins
|
|
will be loaded as well. You can also use dotted path like this::
|
|
|
|
pytest_plugins = "myapp.testsupport.myplugin"
|
|
|
|
which will import the specified module as a ``pytest`` plugin.
|
|
|
|
|
|
Accessing another plugin by name
|
|
--------------------------------------------
|
|
|
|
If a plugin wants to collaborate with code from
|
|
another plugin it can obtain a reference through
|
|
the plugin manager like this:
|
|
|
|
.. sourcecode:: python
|
|
|
|
plugin = config.pluginmanager.getplugin("name_of_plugin")
|
|
|
|
If you want to look at the names of existing plugins, use
|
|
the ``--traceconfig`` option.
|
|
|
|
.. _`findpluginname`:
|
|
|
|
Finding out which plugins are active
|
|
----------------------------------------------------------------------------
|
|
|
|
If you want to find out which plugins are active in your
|
|
environment you can type::
|
|
|
|
py.test --traceconfig
|
|
|
|
and will get an extended test header which shows activated plugins
|
|
and their names. It will also print local plugins aka
|
|
:ref:`conftest.py <conftest>` files when they are loaded.
|
|
|
|
.. _`cmdunregister`:
|
|
|
|
Deactivating / unregistering a plugin by name
|
|
----------------------------------------------------------------------------
|
|
|
|
You can prevent plugins from loading or unregister them::
|
|
|
|
py.test -p no:NAME
|
|
|
|
This means that any subsequent try to activate/load the named
|
|
plugin will it already existing. See :ref:`findpluginname` for
|
|
how to obtain the name of a plugin.
|
|
|
|
.. _`builtin plugins`:
|
|
|
|
pytest default plugin reference
|
|
====================================
|
|
|
|
|
|
You can find the source code for the following plugins
|
|
in the `pytest repository <http://bitbucket.org/hpk42/pytest/>`_.
|
|
|
|
.. autosummary::
|
|
|
|
_pytest.assertion
|
|
_pytest.capture
|
|
_pytest.config
|
|
_pytest.doctest
|
|
_pytest.genscript
|
|
_pytest.helpconfig
|
|
_pytest.junitxml
|
|
_pytest.mark
|
|
_pytest.monkeypatch
|
|
_pytest.nose
|
|
_pytest.pastebin
|
|
_pytest.pdb
|
|
_pytest.pytester
|
|
_pytest.python
|
|
_pytest.recwarn
|
|
_pytest.resultlog
|
|
_pytest.runner
|
|
_pytest.main
|
|
_pytest.skipping
|
|
_pytest.terminal
|
|
_pytest.tmpdir
|
|
_pytest.unittest
|
|
|
|
.. _`well specified hooks`:
|
|
|
|
pytest hook reference
|
|
====================================
|
|
|
|
Hook specification and validation
|
|
-----------------------------------------
|
|
|
|
``pytest`` calls hook functions to implement initialization, running,
|
|
test execution and reporting. When ``pytest`` loads a plugin it validates
|
|
that each hook function conforms to its respective hook specification.
|
|
Each hook function name and its argument names need to match a hook
|
|
specification. However, a hook function may accept *fewer* parameters
|
|
by simply not specifying them. If you mistype argument names or the
|
|
hook name itself you get an error showing the available arguments.
|
|
|
|
Initialization, command line and configuration hooks
|
|
--------------------------------------------------------------------
|
|
|
|
.. currentmodule:: _pytest.hookspec
|
|
|
|
.. autofunction:: pytest_load_initial_conftests
|
|
.. autofunction:: pytest_cmdline_preparse
|
|
.. autofunction:: pytest_cmdline_parse
|
|
.. autofunction:: pytest_namespace
|
|
.. autofunction:: pytest_addoption
|
|
.. autofunction:: pytest_cmdline_main
|
|
.. autofunction:: pytest_configure
|
|
.. autofunction:: pytest_unconfigure
|
|
|
|
Generic "runtest" hooks
|
|
------------------------------
|
|
|
|
All runtest related hooks receive a :py:class:`pytest.Item` object.
|
|
|
|
.. autofunction:: pytest_runtest_protocol
|
|
.. autofunction:: pytest_runtest_setup
|
|
.. autofunction:: pytest_runtest_call
|
|
.. autofunction:: pytest_runtest_teardown
|
|
.. autofunction:: pytest_runtest_makereport
|
|
|
|
For deeper understanding you may look at the default implementation of
|
|
these hooks in :py:mod:`_pytest.runner` and maybe also
|
|
in :py:mod:`_pytest.pdb` which interacts with :py:mod:`_pytest.capture`
|
|
and its input/output capturing in order to immediately drop
|
|
into interactive debugging when a test failure occurs.
|
|
|
|
The :py:mod:`_pytest.terminal` reported specifically uses
|
|
the reporting hook to print information about a test run.
|
|
|
|
Collection hooks
|
|
------------------------------
|
|
|
|
``pytest`` calls the following hooks for collecting files and directories:
|
|
|
|
.. autofunction:: pytest_ignore_collect
|
|
.. autofunction:: pytest_collect_directory
|
|
.. autofunction:: pytest_collect_file
|
|
|
|
For influencing the collection of objects in Python modules
|
|
you can use the following hook:
|
|
|
|
.. autofunction:: pytest_pycollect_makeitem
|
|
.. autofunction:: pytest_generate_tests
|
|
|
|
After collection is complete, you can modify the order of
|
|
items, delete or otherwise amend the test items:
|
|
|
|
.. autofunction:: pytest_collection_modifyitems
|
|
|
|
Reporting hooks
|
|
------------------------------
|
|
|
|
Session related reporting hooks:
|
|
|
|
.. autofunction:: pytest_collectstart
|
|
.. autofunction:: pytest_itemcollected
|
|
.. autofunction:: pytest_collectreport
|
|
.. autofunction:: pytest_deselected
|
|
|
|
And here is the central hook for reporting about
|
|
test execution:
|
|
|
|
.. autofunction:: pytest_runtest_logreport
|
|
|
|
|
|
Debugging/Interaction hooks
|
|
--------------------------------------
|
|
|
|
There are few hooks which can be used for special
|
|
reporting or interaction with exceptions:
|
|
|
|
.. autofunction:: pytest_internalerror
|
|
.. autofunction:: pytest_keyboard_interrupt
|
|
.. autofunction:: pytest_exception_interact
|
|
|
|
|
|
Declaring new hooks
|
|
------------------------
|
|
|
|
Plugins and ``conftest.py`` files may declare new hooks that can then be
|
|
implemented by other plugins in order to alter behaviour or interact with
|
|
the new plugin:
|
|
|
|
.. autofunction:: pytest_addhooks
|
|
|
|
Hooks are usually declared as do-nothing functions that contain only
|
|
documentation describing when the hook will be called and what return values
|
|
are expected.
|
|
|
|
For an example, see `newhooks.py`_ from :ref:`xdist`.
|
|
|
|
.. _`newhooks.py`: https://bitbucket.org/hpk42/pytest-xdist/src/52082f70e7dd04b00361091b8af906c60fd6700f/xdist/newhooks.py?at=default
|
|
|
|
|
|
Using hooks from 3rd party plugins
|
|
-------------------------------------
|
|
|
|
Using new hooks from plugins as explained above might be a little tricky
|
|
because the standard `Hook specification and validation`_ mechanism:
|
|
if you depend on a plugin that is not installed,
|
|
validation will fail and the error message will not make much sense to your users.
|
|
|
|
One approach is to defer the hook implementation to a new plugin instead of
|
|
declaring the hook functions directly in your plugin module, for example::
|
|
|
|
# contents of myplugin.py
|
|
|
|
class DeferPlugin(object):
|
|
"""Simple plugin to defer pytest-xdist hook functions."""
|
|
|
|
def pytest_testnodedown(self, node, error):
|
|
"""standard xdist hook function.
|
|
"""
|
|
|
|
def pytest_configure(config):
|
|
if config.pluginmanager.hasplugin('xdist'):
|
|
config.pluginmanager.register(DeferPlugin())
|
|
|
|
|
|
This has the added benefit of allowing you to conditionally install hooks
|
|
depending on which plugins are installed.
|
|
|
|
|
|
Reference of objects involved in hooks
|
|
===========================================================
|
|
|
|
.. autoclass:: _pytest.config.Config()
|
|
:members:
|
|
|
|
.. autoclass:: _pytest.config.Parser()
|
|
:members:
|
|
|
|
.. autoclass:: _pytest.main.Node()
|
|
:members:
|
|
|
|
.. autoclass:: _pytest.main.Collector()
|
|
:members:
|
|
:show-inheritance:
|
|
|
|
.. autoclass:: _pytest.main.Item()
|
|
:members:
|
|
:show-inheritance:
|
|
|
|
.. autoclass:: _pytest.python.Module()
|
|
:members:
|
|
:show-inheritance:
|
|
|
|
.. autoclass:: _pytest.python.Class()
|
|
:members:
|
|
:show-inheritance:
|
|
|
|
.. autoclass:: _pytest.python.Function()
|
|
:members:
|
|
:show-inheritance:
|
|
|
|
.. autoclass:: _pytest.runner.CallInfo()
|
|
:members:
|
|
|
|
.. autoclass:: _pytest.runner.TestReport()
|
|
:members:
|
|
|