Manual minimal backport from commit e89d23b247.
Fix#12355.
In the issue, it was reported that the `reorder_items` has quadratic (or
worse...) behavior with certain simple parametrizations. After some
debugging I found that the problem happens because the "Fix
items_by_argkey order" loop keeps adding the same item to the deque,
and it reaches epic sizes which causes the slowdown.
I don't claim to understand how the `reorder_items` algorithm works, but
if as far as I understand, if an item already exists in the deque, the
correct thing to do is to move it to the front. Since a deque doesn't
have such an (efficient) operation, this switches to `OrderedDict` which
can efficiently append from both sides, deduplicate and move to front.
* added test case in testing/python/approx.py based on test case provided by reporter in issue #12114
* test cases pass for pytest testing/python/approx.py
* expanded the type annotation to include objects which may cast to a array and renamed other_side to other_side_as_array and asserted that it is not none
Now `importlib` mode will correctly set the imported modules as an attribute of their parent modules.
As helpfully posted on #12194, that's how the Python import module works so we should follow suit.
In addition, we also try to import the parent modules as part of the process of importing a child module, again mirroring how Python importing works.
Fix#12194
Previously we used a hand crafted approach to detect namespace packages, however we should rely on ``importlib`` to detect them for us.
Fix#12112
---------
Co-authored-by: Ran Benita <ran@unusedvar.com>
Creating and initializing the cache directory is interruptible; this
avoids a pathological case where interrupting a cache write can cause
the cache directory to never be properly initialized with its supporting
files.
Unify `Cache.mkdir` with `Cache.set` while I'm here so the former also
properly initializes the cache directory.
Closes#12167.
It makes more sense, also, we have a long term idea of generalizing
fixture support to items defined by other plugins, not just python, in
which case `--fixtures` would definitely not be python-plugin
specific.
Based on pylint's message ``consider-iterating-dictionary`` suggestion.
Surprisingly using a dict or set comprehension instead of a new temp var is
actually consistently slower here, which was not intuitive for me.
```python
from timeit import timeit
families = {1: {"testcase": [1, 2, 3, 5, 8]}}
attrs = {1: "a", 2: "b", 3: "c", 4: "d", 5: "e", 6: "f", 7: "g", 8: "h"}
class Old:
def old(self):
self.attrs = attrs
temp_attrs = {}
for key in self.attrs.keys():
if key in families[1]["testcase"]:
temp_attrs[key] = self.attrs[key]
self.attrs = temp_attrs
class OldBis:
def old(self):
self.attrs = attrs
temp_attrs = {}
for key in self.attrs:
if key in families[1]["testcase"]:
temp_attrs[key] = self.attrs[key]
self.attrs = temp_attrs
class New:
def new(self):
self.attrs = attrs
self.attrs = { # Even worse with k: v for k in self.attrs.items()
k: self.attrs[k] for k in self.attrs if k in families[1]["testcase"]
}
if __name__ == "__main__":
n = 1000000
print(f"Old: {timeit(Old().old, number=n)}")
print(f"Just removing the keys(): {timeit(OldBis().old, number=n)}")
print(f"List comp, no temp var: {timeit(New().new, number=n)}")
```
Result:
Old: 0.9493889989680611
Just removing the keys(): 0.9042672360083088
List comp, no temp var: 0.9916125109884888
It's also true for the other example with similar benchmark, but the exact
code probably does not need to be in the commit message.
The plugin is abandoned and no longer working with new pytest versions.
I also reordered a bit to put pytest-rerunfailures first since it seems most maintained and is under pytest-dev.
and also fixes a regression in pytest 8.0.0 where `setup_method` crashes
if the class has static or class method tests.
It is allowed to have a test class with static/class methods which
request non-static/class method fixtures (including `setup_method`
xunit-fixture). I take it as a given that we need to support this
somewhat odd scenario (stdlib unittest also supports it).
This raises a question -- when a staticmethod test requests a bound
fixture, what is that fixture's `self`?
stdlib unittest says - a fresh instance for the test.
Previously, pytest said - some instance that is shared by all
static/class methods. This is definitely broken since it breaks test
isolation.
Change pytest to behave like stdlib unittest here.
In practice, this means stopping to rely on `self.obj.__self__` to get
to the instance from the test function's binding. This doesn't work
because staticmethods are not bound to anything.
Instead, keep the instance explicitly and use that.
BTW, I think this will allow us to change `Class`'s fixture collection
(`parsefactories`) to happen on the class itself instead of a class
instance, allowing us to avoid one class instantiation. But needs more
work.
Fixes#12065.
- Separate the requesting from the requested.
- Avoid the term "factory", I think most people don't distinguish
between "fixture" and "fixture function" (i.e. "factory") and would
find the term "factory" unfamiliar.
There are two non-optimal things in the current way scope checking is
done:
- It runs on `SubRequest`, but doesn't use the `SubRequest's scope,
which is confusing. Instead it takes `invoking_scope` and
`requested_scope`.
- Because `_check_scope` is only defined on `SubRequest` and not
`TopRequest`, `_compute_fixture_value` first creates the `SubRequest`
only then checks the scope (hence the need for the previous point).
Instead, also define `_check_scope` on `TopRequest` (always valid), and
remove `invoking_scope`, using `self._scope` instead.
Previously, the `obj` of a `TestCaseFunction` (the unittest plugin item
type) was the unbound method. This is unlike regular `Class` where the
`obj` is a bound method to a fresh instance.
This difference necessitated several special cases in in places outside
of the unittest plugin, such as `FixtureDef` and `FixtureRequest`, and
made things a bit harder to understand.
Instead, match how the python plugin does it, including collecting
fixtures from a fresh instance.
The downside is that now this instance for fixture-collection is kept
around in memory, but it's the same as `Class` so nothing new. Users
should only initialize stuff in `setUp`/`setUpClass` and similar
methods, and not in `__init__` which is generally off-limits in
`TestCase` subclasses.
I am not sure why there was a difference in the first place, though I
will say the previous unittest approach is probably the preferable one,
but first let's get consistency.
The check for short paths under Windows via os.path.samefile, introduced in #11936, also found similar tests in symlinked tests in the GH Actions CI.
Fixes#12039.
Co-authored-by: Bruno Oliveira <bruno@soliv.dev>
Previously, if more than one fixture finalizer raised, only the first
was reported, and the other errors were lost.
Use an exception group to report them all. This is similar to the change
we made in node teardowns (in `SetupState`).
As detailed in
https://github.com/pytest-dev/pytest/issues/11475#issuecomment-1937043670,
currently with `--import-mode=importlib` pytest will try to import every
file by using a unique module name, regardless if that module could be
imported using the normal import mechanism without touching `sys.path`.
This has the consequence that non-test modules available in `sys.path`
(via other mechanism, such as being installed into a virtualenv,
PYTHONPATH, etc) would end up being imported as standalone modules,
instead of imported with their expected module names.
To illustrate:
```
.env/
lib/
site-packages/
anndata/
core.py
```
Given `anndata` is installed into the virtual environment, `python -c
"import anndata.core"` works, but pytest with `importlib` mode would
import that module as a standalone module named
`".env.lib.site-packages.anndata.core"`, because importlib module was
designed to import test files which are not reachable from `sys.path`,
but now it is clear that normal modules should be imported using the
standard mechanisms if possible.
Now `imporlib` mode will first try to import the module normally,
without changing `sys.path`, and if that fails it falls back to
importing the module as a standalone module.
This also makes `importlib` respect namespace packages.
This supersedes #11931.
Fix#11475Close#11931
(diff better viewed ignoring whitespace)
In pytest<8, the collection tree for `pyargs` arguments in an invocation
like this:
pytest --collect-only --pyargs pyflakes.test.test_undefined_names
looked like this:
```
<Package test>
<Module test_undefined_names.py>
<UnitTestCase Test>
<TestCaseFunction test_annotationUndefined>
... snipped ...
```
The pytest 8 collection improvements changed it to this:
```
<Dir pytest>
<Dir .tox>
<Dir venv>
<Dir lib>
<Dir python3.11>
<Dir site-packages>
<Package pyflakes>
<Package test>
<Module test_undefined_names.py>
<UnitTestCase Test>
<TestCaseFunction test_annotationUndefined>
... snipped ...
```
Besides being egregious (and potentially even worse than the above,
going all the way to the root, for system-installed packages, as is
apparently common in CI), this also caused permission errors when trying
to probe some of those intermediate directories.
This change makes `--pyargs` arguments no longer try to add parent
directories to the collection tree according to the `--confcutdir` like
they're regular arguments. Instead, only add the parents that are in the
import path. This now looks like this:
```
<Package .tox/venv/lib/python3.11/site-packages/pyflakes>
<Package test>
<Module test_undefined_names.py>
<UnitTestCase Test>
<TestCaseFunction test_annotationUndefined>
... snipped ...
```
Fix#11904.
pytest allows a fixture to request its own name (directly or
indirectly), in which case the fixture with the same name but one level
up is used.
To know which fixture should be used next, pytest keeps a mutable
item-global dict `_arg2index` which maintains this state. This is not
great:
- Mutable state like this is hard to understand and reason about.
- It is conceptually buggy; the indexing is global (e.g. if requesting
`fix1` and `fix2`, the indexing is shared between them), but actually
different branches of the subrequest tree should not affect each
other.
This is not an issue in practice because pytest keeps a cache of the
fixturedefs it resolved anyway (`_fixture_defs`), but if the cache is
removed it becomes evident.
Instead of the `_arg2index` state, count how many `argname`s deep we are
in the subrequest tree ("the fixture stack") and use that for the index.
This way, no global mutable state and the logic is very localized and
easier to understand.
This is slower, however fixture stacks should not be so deep that this
matters much, I hope.
According to my understanding, this code, which handles obtaining the
relevant fixturedefs when a dynamic `getfixturevalue` is used, has an
optimization where it only grabs fixturedefs that are visible to the
*parent* of the item, instead of the item itself, under the assumption
that a fixturedef can't be visible to a single item, only to a
collector.
Remove this optimization for the following reasons:
- It doesn't save much (one loop iteration in `matchfactories`)
- It slightly complicates the complex fixtures code
- If some plugin wants to make a fixture visible only to a single item,
why not let it?
- In the static case (`getfixtureclosure`), this optimization is not
done (despite the confusing name `parentnode`, it is *not* the parent
node). This is inconsistent.
Allow for the output of test case execution to be controlled independently from the application verbosity level.
`verbosity_test_case` is the new ini setting to adjust this functionality.
Fix#11639
Passing a short path in the command line was causing the matchparts check to fail, because ``Path(short_path) != Path(long_path)``.
Using ``os.path.samefile`` as fallback ensures the comparsion works on Windows when comparing short/long paths.
Fix#11895
In `CallSpec2.setmulti` the `params` and `_arg2scope` dicts are always
set together.
Further, the `get_parametrized_fixture_keys` accesses `_arg2scope` for
all argnames without a check, which I think rules out that the code
protects against plugin shenanigans.
After removing the suppress, adjust the comment and code to make more
sense.
Fix#12021.
Reopens#11706.
This reverts commit 12b9bd5801.
This change caused a bad regression in pytest-xdist:
https://github.com/pytest-dev/pytest-xdist/issues/1024
pytest-xdist necessarily has special handling of `--maxfail` and session
fixture teardown get executed multiple times with the change.
Since I'm not sure how to adapt pytest-xdist myself, revert for now.
I kept the sticky `shouldstop`/`shouldfail` changes as they are good
ideas regardless I think.
Dicts these days preserve order, so the sort is no longer needed to
achieve determinism.
As shown by the `test_dynamic_parametrized_ordering` test, this can
change the ordering of items, but only in equivalent ways (same number
of setups/teardowns per scope), it will just respect the user's given
ordering better (hence `vxlan` items now ordered before `vlan` items
compared to the previous ordering).
Follow up to #12006, let's put some comments clarifying `is_in_confcutdir` semantics, as this is not the first time someone misunderstands it.
Also removed an obsolete comment in `_loadconftestmodules`: we already set the `confcutdir` based on `rootdir`/`initfile` if not explicitly given.
Co-authored-by: Ran Benita <ran@unusedvar.com>
As the comment above it states, we need to keep the comparison simple so mypy can understand it, otherwise it will fail to properly handle this on Windows and will flag ``os.getuid()`` missing.
Test:
`warnings.warn()` expects that its first argument is a `str` or a
`Warning`, but since 9454fc38d3
`pytest.warns()` no longer allows `Warning` instances unless the first
argument the `Warning` was initialized with is a `str`. Furthermore, if
the `Warning` was created without arguments then `pytest.warns()` raises
an unexpected `IndexError`. The new tests reveal the problem.
Fix:
`pytest.warns()` now allows using `warnings.warn()` with a `Warning`
instance, as is required by Python, with one exception. If the warning
used is a `UserWarning` that was created by passing it arguments and the
first argument was not a `str` then `pytest.raises()` still considers
that an error. This is because if an invalid type was used in
`warnings.warn()` then Python creates a `UserWarning` anyways and it
becomes impossible for `pytest` to figure out if that was done
automatically or not.
[ran: rebased on previous commit]
Today `pyproject.toml` is the standard for declaring a Python project root, so seems reasonable to consider it for the ini configuration (and specially `rootdir`) in case we do not find other suitable candidates.
Related to #11311
Previously this would trigger an `AssertionError`.
While it could be considered a bug-fix, but given it now can be relied upon, it is probably better to consider it an improvement.
Fix#11311
Should be around 40% faster according to this simplified small benchmark:
python -m timeit "a=[0, 1, 2, 3, 4];b=list((e if i in {0, len(a) -1} else str(e)) for i, e in enumerate(a))"
200000 loops, best of 5: 1.12 usec per loop
python -m timeit "a=[0, 1, 2, 3, 4];b=list((a[0], *(str(e) for e in a[1:-1]), a[-1]))"
500000 loops, best of 5: 651 nsec per loop
python -m timeit "a=[0, 1, 2, 3, 4,5,6,7,8,9,10,11,12,13,14,15,16];b=list((e if i in {0, len(a) -1} else str(e)) for i, e in enumerate(a))"
100000 loops, best of 5: 3.31 usec per loop
python -m timeit "a=[0, 1, 2, 3, 4,5,6,7,8,9,10,11,12,13,14,15,16];b=list((a[0], *(str(e) for e in a[1:-1]), a[-1]))"
200000 loops, best of 5: 1.72 usec per loop
* Improve error message when using @pytest.fixture twice
While obvious in hindsight, this error message confused me. I thought my fixture
function was used in a test function twice, since the wording is ambiguous.
Also, the error does not tell me *which* function is the culprit.
Finally, this adds a test, which wasn't done in
cfd16d0dac where this was originally implemented.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
warning: The top-level linter settings are deprecated in favour of their counterparts in the `lint` section. Please update the following options in `pyproject.toml`:
- 'ignore' -> 'lint.ignore'
- 'select' -> 'lint.select'
Fix#11929.
Figured out what's going on. We have the following collection tree:
```
<Dir pyspacewar>
<Dir src>
<Package pyspacewar>
<Package tests>
<DoctestModule test_main.py>
<DoctestItem pyspacewar.tests.test_main.doctest_main>
```
And the `test_main.py` contains an autouse fixture (`fake_game_ui`) that
`doctest_main` needs in order to run properly. The fixture doesn't run!
It doesn't run because nothing collects the fixtures from (calls
`parsefactories()` on) the `test_main.py` `DoctestModule`.
How come it only started happening with commit
ab63ebb3dc07b89670b96ae97044f48406c44fa0? Turns out it mostly only
worked accidentally. Each `DoctestModule` is also collected as a normal
`Module`, with the `Module` collected after the `DoctestModule`. For
example, if we add a non-doctest test to `test_main.py`, the collection
tree looks like this:
```
<Dir pyspacewar>
<Dir src>
<Package pyspacewar>
<Package tests>
<DoctestModule test_main.py>
<DoctestItem pyspacewar.tests.test_main.doctest_main>
<Module test_main.py>
<Function test_it>
```
Now, `Module` *does* collect fixtures. When autouse fixtures are
collected, they are added to the `_nodeid_autousenames` dict.
Before ab63ebb3dc, `DoctestItem` consults
`_nodeid_autousenames` at *setup* time. At this point, the `Module` has
collected and so it ended up picking the autouse fixture (this relies on
another "accident", that the `DoctestModule` and `Module` have the same
node ID).
After ab63ebb3dc, `DoctestItem` consults
`_nodeid_autousenames` at *collection* time (= when it's created). At
this point, the `Module` hasn't collected yet, so the autouse fixture is
not picked out.
The fix is simple -- have `DoctestModule.collect()` call
`parsefactories`. From some testing I've done it shouldn't have negative
consequences (I hope).
Currently, `DoctestModule` does `import_path` on its own. This changes
it to use `importtestmodule` as `Module` does. The behavioral changes
are:
- Much better error messages on import errors.
- Handles a few more error cases (see `importtestmodule`). This
technically expands the cover of `--doctest-ignore-import-errors` but
I think it makes sense.
- Considers `pytest_plugins` in the module.
- Populates `self.obj` as properly (without double-imports) as is
expected from a `PyCollector`.
This is also needed for the next commit.
ruff is faster and handle everything we had prior.
isort configuration done based on the indication from
https://github.com/astral-sh/ruff/issues/4670, previousely based on
reorder-python-import (#11896)
flake8-docstrings was a wrapper around pydocstyle (now archived) that
explicitly asks to use ruff in https://github.com/PyCQA/pydocstyle/pull/658.
flake8-typing-import is useful mainly for project that support python 3.7
and the one useful check will be implemented in https://github.com/astral-sh/ruff/issues/2302
We need to keep blacken-doc because ruff does not handle detection
of python code inside .md and .rst. The direct link to the repo is
now used to avoid a redirection.
Manual fixes:
- Lines that became too long
- % formatting that was not done automatically
- type: ignore that were moved around
- noqa of hard to fix issues (UP031 generally)
- fmt: off and fmt: on that is not really identical
between black and ruff
- autofix re-order in pre-commit from faster to slower
Co-authored-by: Ran Benita <ran@unusedvar.com>
The current version (0.23.4) explicitly does not support pytest 8 yet, so we fallback to the previous release in the hope that at least our integration tests pass.
- Previously, `getpass.getuser()` would leak an ImportError if the
USERNAME environment variable was not set on Windows because the `pwd`
module cannot be imported.
- Starting in Python 3.13.0a3, it only raises `OSError`.
Fixes#11874
Change our mypy configuration to disallow untyped defs by default, which ensures *new* files added to the code base are fully typed.
To avoid having to type-annotate everything now, add `# mypy: allow-untyped-defs` to files which are not fully type annotated yet.
As we fully type annotate those modules, we can then just remove that directive from the top.
The function currently uses `find_suffixes` which iterates the entire
directory searching for files with the given suffix. In some cases
though, like in pytest's selftest, the directory can get big:
$ ls /tmp/pytest-of-ran/pytest-0/
7686
and iterating it many times can get slow.
This doesn't fix the underlying issue (iterating the directory) but at
least speeds it up a bit by using `os.scandir` instead of
`path.iterdir`. So `make_numbered_dir` is still slow for pytest's
selftests, but reduces ~10s for me.
Follow up to https://github.com/pytest-dev/pytest/pull/11754, noticed that the latest GitHub release does not contain the attached files.
Output log from the action:
```
🤔 Pattern 'dist/*' does not match any files.
```
We have a use case for this in the next commit.
The name can be obtained by using `manager.get_name(plugin)`, however
this is currently O(num plugins) in pluggy, which would be good to
avoid. Besides, it seems generally useful.
This removes one thing that directory collectors need to worry about.
This adds one hook dispatch per `__pycache__` file, but I think it's
worth it for consistency.
(Diff better viewed ignoring whitespace)
Since e1c66ab0ad, conftest loading is
handled at the directory level before sub-nodes are collected, so there
is no need for the doctest plugin to handle it specially.
This was probably the case even before
e1c66ab0ad, but I haven't verified this.
This is a useful addition to the existing `listchain`. While `listchain`
returns top-to-bottom, `iterparents` is bottom-to-top and doesn't require
an internal full iteration + `reverse`.
Since we're removing nose support, let's also drop support for this
attribute.
From doing a code search on github, this seems completely unused outside
of nose, except for some projects which used to use it, but no longer
do.
Refs #11662.
--- Problem
Each fixture definition has a "visibility", the `FixtureDef.baseid`
attribute. This is nodeid-like string. When a certain `node` requests a
certain fixture name, we match node's nodeid against the fixture
definitions with this name.
The matching currently happens on the *textual* representation of the
nodeid - we split `node.nodeid` to its "parent nodeids" and then check
if the fixture's `baseid` is in there.
While this has worked so far, we really should try to avoid textual
manipulation of nodeids as much as possible. It has also caused problem
in an odd case of a `Package` in the root directory: the `Package` gets
nodeid `.`, while a `Module` in it gets nodeid `test_module.py`. And
textually, `.` is not a parent of `test_module.py`.
--- Solution
Avoid this entirely by just checking the node hierarchy itself. This is
made possible by the fact that we now have proper `Directory` nodes
(`Dir` or `Package`) for the entire hierarchy.
Also do the same for `_getautousenames` which is a similar deal.
The `iterparentnodeids` function is no longer used and is removed.
Instead of modifying user objects like modules and classes that we
really shouldn't be touching, use the new `_register_fixture` internal
API to do it directly.
Add a function on the `FixtureManager` to register a fixture with
pytest. Currently this can only be done through `parsefactories`.
My aim is to eventually make something like this available to plugins,
as it's a pretty common need.
* Fix handling empty values of NO_COLOR and FORCE_COLOR
Fix handling NO_COLOR and FORCE_COLOR environment variables to correctly
be ignored when they are set to an empty value, as defined
in the specification:
> Command-line software which adds ANSI color to its output by default
> should check for a NO_COLOR environment variable that, when present
> *and not an empty string* (regardless of its value), prevents
> the addition of ANSI color.
(emphasis mine, https://no-color.org/)
The same is true of FORCE_COLOR, https://force-color.org/.
* Streamline testing for FORCE_COLOR and NO_COLOR
Streamline the tests for FORCE_COLOR and NO_COLOR variables, and cover
all possible cases (unset, set to empty, set to "1"). Combine the two
assert functions into one taking boolean parameters. Mock file.isatty
in all circumstances to ensure that the environment variables take
precedence over the fallback value resulting from isatty check (or that
the fallback is actually used, in the case of both FORCE_COLOR
and NO_COLOR being unset).
* Put a 'reset' color in front of the highlighting
When doing the highlighting, some lexers will not set the initial color
explicitly, which may lead to the red from the errors being propagated
to the start of the expression
* Add syntactic highlighting to the error explanations
This updates the various error reporting to highlight python code when
displayed, to increase readability and make it easier to understand
--- Current main
In current main (before pervious commit), calls to gethookproxy/ihook
are the trigger for loading non-initial conftests. This basically means
that conftest loading is done almost as a random side-effect,
uncontrolled and very non-obvious. And it also dashes any hope of making
gethookproxy faster (gethookproxy shows up prominently in pytest
profiles).
I've wanted to improve this for a while, #11268 was the latest step
towards that.
--- PR before change
In this PR, I ran into a problem.
Previously, Session and Package did all of the directory traversals
inside of their collect, which loaded the conftests as a side effect. If
the conftest loading failed, it will occur inside of the collect() and
cause it to be reported as a failure.
Now I've changed things such that Session.collect and Package.collect no
longer recurse, but just collect their immediate descendants, and
genitems does the recursive expansion work.
The problem though is that genitems() doesn't run inside of specific
collector's collect context. So when it loads a conftest, and the
conftest loading fails, the exception isn't handled by any CollectReport
and causes an internal error instead.
The way I've fixed this problem is by loading the conftests eagerly in a
pytest_collect_directory post-wrapper, but only during genitems to make
sure the directory is actually selected.
This solution in turn caused the conftests to be collected too early;
specifically, the plugins are loaded during the parent's collect(), one
after the other as the directory entries are collected. So when the
ihook is hoisted out of the loop, new plugins are loaded inside the
loop, and due to the way the hook proxy works, they are added to the
ihook even though they're supposed to be scoped to the child collectors.
So no hoisting.
--- PR after change
Now I've come up with a better solution: since now the collection tree
actually reflects the filesystem tree, what we really want is to load
the conftest of a directory right before we run its collect(). A
conftest can affect a directory's collect() (e.g. with a
pytest_ignore_collect hookimpl), but it cannot affect how the directory
node itself is collected. So I just moved the conftest loading to be
done right before calling collect, but still inside the CollectReport
context.
This allows the hoisting, and also removes conftest loading from
gethookproxy since it's no longer necessary. And it will probably enable
further cleanups. So I'm happy with it.
Previously the error report would have all sections glued together:
- The assertion representation
- The error explanation
- The full diff
This makes it hard to see at a glance where which one starts and ends.
One of the representation (dataclasses, tuples, attrs) does display a
newlines at the start already.
Let's add a newlines before the error explanation and before the full
diff, so we get an easier to read report.
This has one disadvantage: we get one line less in the least verbose
mode, where the output gets truncated.
The default for `_prepareconfig` is to use `sys.argv`, which in this
case are the flags passed to (top-level) `pytest`. This is not the
intention, the tests themselves should not be affected by it.
When running `tox -e py-lsof` I get a deluge of this warning:
```
src/pytest/.tox/py-lsof-numpy-pexpect/lib/python3.11/site-packages/_pytest/pytester.py:130: EncodingWarning: UTF-8 Mode affects locale.getpreferredencoding(). Consider locale.getencoding() instead.
```
Use `locale.getencoding` instead.
The normal default pretty printer is not great when objects are nested
and it can get hard to read the diff.
Instead, provide a pretty printer that behaves more like when json get
indented, which allows for smaller, more meaningful differences, at
the expense of a slightly longer diff.
This does not touch the other places where the pretty printer is used,
and only updated the full diff one.
Reset color-related environment variables in a fixture to prevent them
from affecting test results. Otherwise, some of the tests fail
e.g. if NO_COLOR is set in the calling environment.
We already have the AlwaysDispatchingPrettyPrinter override of the
default pretty printer. In order to make more in depth changes, we
need to copy the upstream version in, as it doesn't lend itself well to
being extended.
This does a verbatime copy, adding provenance information at the top.
if we were ever to change the `name:` field in `test.yml` this would suddenly break -- this references the workflow file directly (and is what github generates now for badges)
The tested py.path.local.make_numbered_dir function is *not*
multiprocess safe, because is uses os.listdir which itself is not.
The os.listdir documentation explicitly states that:
> If a file is removed from or added to the directory during the call
> of this function, whether a name for that file be included is unspecified.
This can lead to a race when:
1. process A attempts to create directory N
2. the creation fails, as another process already created it in the meantime
3. process A calls listdir to determine a more recent maxnum
4. processes B+ repeatedly create newer directories and they delete directory N
5. process A doesn't have directory N or any newer directory in listdir result
6. process A attempts to create directory N again and raises
For details, see https://github.com/pytest-dev/pytest/issues/11603#issuecomment-1805708144
and bellow.
Additionally, the test itself has a race in batch_make_numbered_dirs.
When this functions attempts to write to repro-N/foo,
repro-N may have already been removed by another process.
For details, see https://github.com/pytest-dev/pytest/issues/11603#issuecomment-1804714313
and bellow.
---
The tested py.path.local.make_numbered_dir function is not used in pytest.
There is a different implementation in _pytest.pathlib.
Closes#11603
Previously, the sidebar TOC had unlimited depth, making it useless and
interfering with the content. This seems to have regressed in pytest
7.2.x going by the RTD version selector.
Traditionally pytest publishes tags in the form `X.Y.Z`, however the deploy workflow (copied from somewhere else) published tags in the form `vX.Y.Z`.
This is the root cause of #11548, because it tried to publish the release notes for tag `X.Y.Z` (which did not exist).
Fix#11548
Logging has many global states, and we did foresee this by creating a ``cleanup_disabled_logging`` fixture,
however one might still forget to use it and failures leak later -- sometimes not even in the same PR, because the order
of the tests might change in the future, specially when running under xdist.
This problem surfaced during pytest-dev/pytest#11530, where tests unrelated to the change started to fail.
The current method as the following problem, described by Sadra
Barikbin:
The tests that request both `pytester` and `monkeypatch` and use
`monkeypatch.chdir` without context, relying on `monkeypatch`'s teardown
to restore cwd. This doesn't work because the following sequence of
actions take place:
- `monkeypatch` is set up.
- `pytester` is set up. It saves the original cwd and changes it to a
new one dedicated to the test function.
- Test function calls `monkeypatch.chdir()` without context.
`monkeypatch` saves cwd, which is not the original one, before
changing it.
- `pytester` is torn down. It restores the cwd to the original one.
- `monkeypatch` is torn down. It restores cwd to what it has saved.
The solution here is to have pytester use `monkeypatch.chdir()` itself,
then everything is handled correctly.
Tests if a captured exception group contains an expected exception.
Will raise `AssertionError` if the wrapped exception is not an exception group.
Supports recursive search into nested exception groups.
Starting with `resolve_package_path` and its associated tests,
this refactoring seeks to make path concatenation more
readable and consistent within tests/functions.
As discussed in #11413:
- code is free to use either `/` and `joinpath`
- consistency within a function is more important than consistency across the codebase
- it is nice to use `/` when it is more readable
- it is nice to use `joinpath` when there is little context
- be mindful that `joinpath` may be clearer when joining multiple segments
There used to be two callers to `_setup_fixtures()`, now there's only
one, so inline it and make `DoctestItem` more similar to `Function`.
(Eventually we may want to generalize `TopRequest` from taking
`Function` directly to some "fixture-supporting item", removing the
remaining `type: ignore` here and allowing plugins to do it in a stable
manner).
As discussed in #11408:
* Improve documentation for the release process.
* Fix the description for the PRs created by the `prepare release pr` workflow.
* Fix pushing tag in the `deploy` workflow.
For packages, `import_path` receives the path to the package's `__init__.py` file, however module names (as they live in `sys.modules`) should not include the `__init__` part.
For example, `app/core/__init__.py` should be imported as `app.core`, not as `app.core.__init__`.
Fix#11306
I could find 2 plugins that would be broken by this (pytest-play and
pytest-wdl), but they will be better served by just copying
`_fillfixtures` instead of use the private function.
`_pyfuncitem.fixturenames` is just an alias for
`_pyfuncitem._fixtureinfo.names_closure` (at least in core pytest), so
let's do the less abstraction-breaking thing.
Move handling of user_properties to `finalize()`.
Previously if a fixture failed during teardown, `pytest_runtest_logreport` would not be called with "teardown", resulting in the user properties not being saved on the JUnit XML file.
Fixes: #11367
* Build the package only once, and test on all platforms.
* Deploy is now triggered manually via an Action, which is then responsible for tagging the repository after the package has been uploaded successfully.
* Drop 'docs': we nowadays rely on readthedocs preview PR builds.
This reverts commit e938580257.
Revert "improve error msg and test"
This reverts commit c0cf822ca1.
Revert "error msg"
This reverts commit ec1053cc16.
Revert "changelog"
This reverts commit d2dc8a70b5.
Revert "simplify code / take out user-gen typeerror case"
This reverts commit b9cb87d862.
Setting `norecursedirs` overrides the default, so we end up scanning
dot-directories and such which slows down collection unnecessarily
(150ms on my working directory).
To remove fixtures.py::add_funcargs_pseudo_fixture_def and add its logic
i.e. registering funcargs as params and making corresponding fixturedefs,
right to Metafunc.parametrize in which parametrization takes place.
To remove funcargs from metafunc attributes as we populate metafunc
params and make pseudo fixturedefs simultaneously and there's no need to
keep funcargs separately.
Previously, when assigning a scope for a fully-indirect parameter set,
when there are multiple fixturedefs for a param (i.e. same-name fixture
chain), the highest scope was used, but it should be the lowest scope,
since that's the effective scope of the fixture.
What one typically actually wants in such a case is both, checking for some
resulting values *and* checking for some expected exception.
Since this is easily possible with the `nullcontext` context manager, adapt the
example accordingly (needlessly using a different name rather just confuses people).
Signed-off-by: Christoph Anton Mitterer <mail@christoph.anton.mitterer.name>
Previously, the `_getconftestmodules` function was used both to load
conftest modules for a path (during `pytest_load_initial_conftests`),
and to retrieve conftest modules for a path (during hook dispatch and
for fetching `collect_ignore`). This made things muddy - it is usually
nicer to have clear separation between "command" and "query" functions,
when they occur in separate phases.
So split into "load" and "get".
Currently, `gethookproxy` still loads conftest itself. I hope to change
this in the future.
this change
* switches from html index usage to json index usage
* adds using requests-cache as http cache to trim down runtime
* adds gh action caches for pip and the new http cache
This test makes clear the need for the `_check_scope()` call in the
`pytest_setup_fixture` impl in fixtures.py, which otherwise seems
redundant with the one in `_compute_fixture_value`.
FixtureDef is used in the `pytest_fixture_setup` hook so needs to be
public. However since its current internals are quite dubious (and not
all marked with `_` prefix) I've added an explicit note that only
documented fields/methods are considered public.
Refs #7469.
From understanding the code better I see this is the correct fix.
The fixturedefs can be None if `request.getfixturevalue("doesnotexist")`
is used.
In practice there is no change in behavior because this mapping is used
as `self._arg2fixturedefs.get(argname, None)` which ends up the same.
It makes for a more debuggable repr. Before:
<function FixtureRequest._schedule_finalizers.<locals>.<lambda> at 0x7fe4ae32d440>
After:
functools.partial(<bound method FixtureDef.finish of <FixtureDef argname='hello_package' scope='package' baseid=''>>, request=<SubRequest 'hello_package' for <Function test_hello>>)
Add explicit mentions of the default behavior to both API reference and
how-to section about ways to invoke pytest.
Co-authored-by: Ran Benita <ran@unusedvar.com>
Dict comparsion in the ApproxMapping class did not check if values were None before attempting to subtract for max_abs_diff stat, which was throwing an TypeError instead of being handled by pytest error assertion. Check for None has been added before these calculations, so that None will properly show as Obtained/Expected in pytest assert message
The initial implementation (in #7246) introduced the `importlib` mode, which
never added the imported module to `sys.modules`, so it included a test
to ensure calling `import_path` twice would yield different modules.
Not adding modules to `sys.modules` proved problematic, so we began to add the imported module to `sys.modules`
in #7870, but failed to realize that given we are now changing `sys.modules`, we might
as well avoid importing it more than once.
Then #10088 came along, passing `importlib` also when importing application modules
(as opposed to only test modules before), which caused problems due to imports
having side-effects and the expectation being that they are imported only once.
With this PR, `import_path` returns the module immediately if already in
`sys.modules`.
Fix#10811Fix#10341
Fixes#11104.
See the issue for a description of the problem.
Now, we use the same logic for initial conftest paths as we do for
deciding the initial args, which was the idea behind checking
`namespace.file_or_dir` and `testpaths` previously.
This fixes the issue of `testpaths` being considered for initial
conftests even when it's not used for the args.
(Another issue in faeb16146b was that the
`testpaths` were not glob-expanded, this is also fixed.)
I think it's helpful to separate the node classes from the other
objects, as they have their own unique usage.
I've chosen not to alphabetize the order, but to use a logical order
instead.
Also slightly improve the docstrings.
Currently, if `--confcutdir` is not set, `inipath.parent` is used, and
if `initpath` is not set, then `confcutdir` is None, which means there
is no cutoff.
Having no cutoff is not great, it means we potentially start probing
stuff all the way up to the filesystem root directory. So let's add
another fallback, to `rootpath`, which is always something reasonable.
In #10758 we introduced the support for the use of the walrus operator in the test cases. There was a case which was not handled that caused a bug report #11028. This PR aims to fix the issue and also to improve how the walrus operator is handled in the AssertionRewriter class.
Closes#11028
Added handling of %f directive to print microseconds in log format options, such as log-date-format. It is impossible to do with a standard logging.Formatter because it uses time.strftime which doesn't know about milliseconds and %f. In this PR I added a custom Formatter which converts LogRecord to a datetime.datetime object and formats it with %f flag. This behaviour is enabled only if a microsecond flag is specified in a format string.
Also added a few tests to check the standard and changed behavior.
Closes#10991
`--lf` has a feature where if a certain `Module` (python file) does not
contain any failed tests, it is skipped entirely at the collector level
instead of being collected and each item skipped individually. When this
happens the collection summary looks like this:
run-last-failure: rerun previous 1 failure (skipped 1 file)
However, this feature didn't work for `Module`s inside of `Package`s,
only for those directly beneath the `Session`.
Fix#11054.
The definition of `repr_failure` on the `YamlItem` subclass only handled the custom `YamlException` class, which hides all other errors from the user. By adding in the `super` call we ensure all other exception types also appropriately handled by `repr_failure`.
Forces requested `caplog` logging levels to be enabled if they were disabled via `logging.disable()`
`[attr-defined]` mypy error ignored in `logging.py` because there were existing errors with the imports
and `loggin.Logger.manager` is an attr set at runtime. Since it's in the standard lib I can't really fix that.
Ignored an attr-defined error in `src/_pytest/config/__init__.py` because the re-export is necessary.
Fixes#8711
`_set_initial_conftests` could break on some systems if a very long
option was passed, because the `Path.exists()` call raises an
`OSError` instead of returning `False`.
Fix#10169
This introduces a workflow to automatically close issues with the label
`status: needs information` after a number of days of inactivity.
This work has been done manually for a number of years, but I think it is safe
to close issues with this label automatically.
Not tested yet, but it is in `debug-only` mode so we can watch what it does before
deciding to turn it on (however it needs to be in `main` for it to run).
Fixes a pytest-xdist regression after
762bb61562 (not yet released).
pytest-xdist patches sys.stderr with an object which doesn't have
`encoding`. Strictly speaking, this should be fixed there (or more
precisely, in execnet), but it will drop support for older versions
which don't want.
But in any case, the fix turns out to simplify the code, using FD
support added in Python 3.5, so it's good anyway!
Refs: https://github.com/pytest-dev/pytest-xdist/pull/900
* faulthandler: fix an EncodingWarning
* _py/path: tiny change to `ensure` to silence EncodingWarning
We're not supposed to diverge here, but make this change to fix an
unavoidable EncodingWarning that is otherwise raised in pytest's test
suite. The behavior should be exactly the same besides the warning,
hopefully that won't cause confusion.
Since `Traceback.getcrashentry` takes the `ExceptionInfo`, it is not
really independent of it and is in the wrong layer. Prevent nonsensical
mistakes by inlining it.
TracebackEntry needs the excinfo for the `__tracebackhide__ = callback`
functionality, where `callback` accepts the excinfo.
Currently it achieves this by storing a weakref to the excinfo which
created it. I think this is not great, mixing layers and bloating the
objects.
Instead, have `ishidden` (and transitively, `Traceback.filter()`) take
the excinfo as a parameter.
Closes#10831.
This fixes a small bug where running tests that contained
`pytest.fail(pytrace=False)` with the `--tb=line` flag set results in
an output of "None" in the Failures section of the output, and adds
a test to ensure the behavior is correct.
The problem is that we would loop over all directories of the basetemp directory searching for dead symlinks, for each test, which would compound over the test session run.
Doing the cleanup just once, at the end of the session, fixes the problem.
Fix#10896
By "empty traceback" I mean a traceback all of whose entries have been
filtered/cut/pruned out.
Currently, if an empty traceback needs to be repr'ed, the last entry
before the filtering is used instead (added in
accd962c9f).
Showing a hidden frame is not so good IMO. This commit does the
following instead:
1. Shows details of the exception.
2. Shows a message about how the full trace can be seen.
Example:
```
_____________ test _____________
E ZeroDivisionError: division by zero
All traceback entries are hidden. Pass `--full-trace` to see hidden and internal frames.
```
Also handles `--tb=native`, though there the `--full-trace` bit is not
shown.
This commit contains some pieces from
431ec6d34e (which has been reverted).
Helps towards fixing issue # 1904.
Co-authored-by: Felix Hofstätter <Felhof1@hotmail.com>
The old-style `sys.exc_info()` triplet is redundant nowadays with
`(type(exc), exc, exc.__traceback__)`, and is beginning to get
soft-deprecated in Python 3.12.
Add a nicer API to ExceptionInfo which takes just the exc instead of the
triplet. There are already a few internal uses which benefit.
Fix#10875
Without this, fails with
```
...
E AttributeError: 'TestCaseFunction' object has no attribute 'addDuration'
...
E RuntimeWarning: TestResult has no addDuration method
```
Updated the c fixture to be a little more consistent with other fixtures in the corresponding image. for example both e and g both have edges connected with the fixtures that they explicitly depend on.
Nowadays, the prepend and append import modes use importlib.import_module() instead of __import__().
There was a phrase “which avoids having to use `__import__`”, in which I couldn’t just replace `__import__` by `importlib.import_module` because the latter is used (in insert_missing_modules()) also when using importlib mode. Therefore I removed the part from the sentence.
This makes it possible to correlate pytest stages with external events, and also makes it readable when TestReports are exported externall (for example with pytest-reportlog).
Closes#10710
Instead of `SysCapture`/`FDCapture` inheriting from
`SysCaptureBinary`/`FDCaptureBinary`, have both inherit from a common
`SysCaptureBase`/`FDCaptureBase`. This fixes a Liskov substitution
violation.
Since pytest now requires Python>=3.7, we can use the stdlib attrs
clone, dataclasses, instead of the OG package.
attrs is still somewhat nicer than dataclasses and has some extra
functionality, but for pytest usage there's not really a justification
IMO to impose the extra dependency on users when a standard alternative
exists.
Makes `AbstractContextManager` the shared base class between "raises" and other context managers.
The motivation is for type checkers to narrow `pytest.raises(...) if x else nullcontext()` to a `ContextManager` rather than `object`.
Fixes https://github.com/pytest-dev/pytest/issues/10604 which could intermittently display unexpected behavior between checking if the path exists and requesting creation. This was fairly prevalent when pytest was being invoked in parallel by another test runner (CTest) and trying to create the same parent-folder for multiple XMLs. A modest amount of testing did not reproduce other filesystem race conditions.
This notably does not work around an edge case where the parent path of the XML could be created as a file instead of a folder or link. That vanishingly rare case should cause file creation to fail on the next line, with a fairly obvious exception message.
The latest 'packaging' release has dropped support for `LegacyVersion`, meaning
that version strings which do not conform to the standard now raise an error.
Packaging >=22 no longer supports 'latest' versions, which is the version that is assigned when building the docs.
https://github.com/pytest-dev/pytest/pull/10578#issuecomment-1348249045
Also first install pytest, then `docs/en/requirements.txt`:
Installing the requirements last will honor pins of libraries
shared between pytest and the docs build.
Closes#10473
Python <3.11 versions depend on `exceptiongroup>=1.0.0rc8`, and they released version `1.0.1`
6 days ago (2022/11/03) that as a side-effect changed the output of exceptions.
Otherwise we see this warning:
Warning: You are using "pypa/gh-action-pypi-publish@master". The "master" branch of this project has been sunset and will not receive any updates, not even security bug fixes. Please, make sure to use a supported version. If you want to pin to v1 major version, use "pypa/gh-action-pypi-publish@release/v1". If you feel adventurous, you may opt to use use "pypa/gh-action-pypi-publish@unstable/v1" instead. A more general recommendation is to pin to exact tags or commit shas.
Tags from the main repository are required for testing to work properly, but the current documentation does not reflect this, and at least one new contributor has been tripped up by this. Update the documentation with a short explanation and directions.
mypy stubs recently changed warnings.catch_warnings to a Generic, in
order to have proper overloads depending on the parameters passed to it, whihc
triggers this mypy error now when we subclass it:
src/_pytest/recwarn.py:170: error: Missing type parameters for generic type "catch_warnings" [type-arg]
For our porpuses the parameter is not relevant (we always use record=True), so
decided to just ignore the type error.
The builtin actions token is not allowed to trigger further workflows.
So when we create a PR using that token, the PR checks don't run, and a
maintainer must close & open the PR to make them run. This is quite
annoying, so switch back to the "machine user" approach.
Fixes#9554.
* Recommend importlib import mode for new projects
* Recommend src layout more strongly
* Switch to hatchling as the packaging tool in the example (following PyPA)
* Add explanation about the different import modes
It is unnecessary and has been deleted from the setuptools' docs too.
The setuptools' PEP 517 build backend implementation has been
auto-adding the `wheel` dependency since it's first been implemented.
* Squashed commit of the following:
commit 41d339c46763bbe26123e1e6504b6e32290e33e1
Author: Cheukting <cheukting.ho@gmail.com>
Date: Thu Jun 23 17:01:04 2022 +0800
test in all py versions
commit b3572a5a12672228c3276fc8c8e05980dfb7888a
Author: Cheukting <cheukting.ho@gmail.com>
Date: Thu Jun 23 16:41:06 2022 +0800
add test
commit 7166a2a51e4f99046b028b663c193d8b558c7fd4
Author: Cheukting <cheukting.ho@gmail.com>
Date: Thu Jun 23 16:00:07 2022 +0800
update changelog
commit b958c73d489157f0c0d4e46425083a5e2e2bc851
Author: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Date: Thu Jun 23 07:50:52 2022 +0000
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
commit ea7f376c6ca37c40c83df0e4a1cfaaedb34bae91
Author: Cheukting <cheukting.ho@gmail.com>
Date: Thu Jun 23 15:48:21 2022 +0800
Fix MyPy
commit 97469beb1da40257e9a061a5e19548546c9312c4
Author: Cheukting <cheukting.ho@gmail.com>
Date: Thu Jun 23 15:03:48 2022 +0800
fix if ExceptionGroup not exist
commit 84e553642cd69b4d499231d733df91ebfa84c7ad
Author: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Date: Thu Jun 23 03:43:27 2022 +0000
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
commit 76bbef449b88bbd74fb5cca3b5293337a624ef03
Author: Cheukting <cheukting.ho@gmail.com>
Date: Thu Jun 23 11:40:41 2022 +0800
adding changelog
commit db82bebc5a4969e2083adcd97bdfd2a63bb17d98
Author: Cheukting <cheukting.ho@gmail.com>
Date: Thu Jun 23 11:33:10 2022 +0800
fall back to native when handeling to exception groups
* Typed ExceptionGroupTypes and changed to BaseExceptionGroup, fixed exceptionchain (excinfo->excinfo_, set reprcrash. Extended tests, though they're wip.
* added exceptiongroup to pre-commit-config, moved away from tuple to directly defining BaseExceptionGroup, added block comment, added match line for inner exception, changked mark.skipif to importorskip to not need top-level import, changed tox.ini a bit - only uncovered should now be py37 without exceptiongroup, due to hypothesis
* added py311-exceptiongroup to github CI, exceptiongroup is now a hard dependency on py<3.11, renamed bad variable names
* added use_coverage to ubuntu-py311
* import BaseExceptionGroup with explicit version check instead of try/catch
* removed from CI, added comments to tox and pre-commit
While taking a look at the docs I noticed they were not properly sorted.
This just moves the sections around so they are sorted alphabetically, without any other changes.
This should help users with the common issue of patching the wrong place.
Also took the opportunity of using proper links in the monkeypatch introduction.
Related to #10216
Co-authored-by: Ran Benita <ran@unusedvar.com>
The main motivation for this change is to simplify the type shown in
code editors -- `Sequence[str]` is easier to follow than
`Union[list[str], tuple[str, ...]]`.
It also permits using other types if desired. It might lead to problems
if someone uses some oddball sequence type, but hopefully they won't do
that.
New Sphinx added support for overloads and always displays them all with
full type annotations etc. This regresses the API reference for
overloaded functions like `fixture()`, `warns()`, `raises()` and friends
to become impossible to read.
I tried various workarounds but none worked except this one.
New versions of sphinx starting showing `__init__` parameters even when
we don't want them to show because they are private (have `_ispytest`
argument).
The only working solution I found was to switch to
`autodoc_typehints_description_target = "documented"` and explicitly
document parameters for which we want to show the types. It's a little
tedious and repetitive in some simple cases, but overall it results in
nicer API docs.
This started to fail recently with:
```
mypy.....................................................................Failed
- hook id: mypy
- exit code: 1
src/_pytest/assertion/rewrite.py:284: error: Returning Any from function declared to return "TraversableResources" [no-any-return]
Found 1 error in 1 file (checked 219 source files)
```
Not sure why that started failing, but seems like ignoring that error specifically is OK.
Given we are already creating a `PurePath`, just pass the parts directly to it.
This avoids using `os.path.sep`, that although is an official API, seems not to be available in all systems.
Fix#9791
`usage.rst` has been moved from `doc/en` to `doc/en/how-to`, so the `man_pages` configuration
value needs to be updated to the new location, so that we dont get this warning:
writing... WARNING: "man_pages" config value references unknown document usage
`usage.rst` has been moved from `doc/en` to `doc/en/how-to`, so the `man_pages` configuration
value needs to be updated to the new location, so that we dont get this warning:
writing... WARNING: "man_pages" config value references unknown document usage
- Remove the index page https://docs.pytest.org/en/7.1.x/genindex.html
Such an index is reasonable in books but not so much in HTML pages.
IMHO the integrated search is much more user-friendly and sufficient.
- Remove the PDF link from the entry page. The PDF is already linked
in the sidebar. I anticipate that getting the PDF documentation is
not a major concern for users.
- Remove the sections "Changelog" and "License" from the entry page.
These section only contain links to the respective pages.
These topics are purely technical. They should be easily accessible
(which they still are through sidebar section "About the project"),
but they don't need additional advertisement through a dedicated
section on the entry page.
This is especially useful for large repositories (e.g. monorepos) that
use a hierarchical file system organization for nested test paths.
src/*/tests
The implementation uses the standard `glob` module to perform wildcard
expansion in Config.parse().
The related logic that determines whether or not to include 'testpaths'
in the terminal header was previously relying on a weak heuristic: if
Config.args matched 'testpaths', then its value was printed. That
generally worked, but it could also print when the user explicitly used
the same arguments on the command-line as listed in 'testpaths'. Not a
big deal, but it shows that the check was logically incorrect.
Now that 'testpaths' can contain wildcards, it's no longer possible to
perform this simple comparison, so this change also introduces a public
Config.ArgSource enum and Config.args_source attribute that explicitly
names the "source" of the arguments: the command line, the invocation
directory, or the 'testdata' configuration value.
This changes the order to:
- API Reference
- Fixtures reference
- Configuration
- Exit codes
- Plugin List
which is approximately sorted from general to specific and often used to less used.
Plugin List ist at the end because it points to further external resources.
This extensive explanation is not need anymore because they are now
unsupported for a long time.
Instead add as short section on Python version compatibility in the
backward compatibility docs.
It's better to have the documentation in one place, instead
of having some in the docstring and some additional
information added to the reference documentation in
`reference.rst`.
Up to now, some had the prefix, some didn't. I think it's good to have
this prefix to give more context, otherwise the links sometimes blend
into the description.
In one case, the link goes to an example, so I used `**Example**:` there.
Up to now, some had the prefix, some didn't. I think it's good to have
this prefix to give more context, otherwise the links sometimes blend
into the description.
In one case, the link goes to an example, so I used `**Example**:` there.
These pytester utility methods were annotated to only receive `str`
names, but they naturally support os.PathLike values, as well.
This makes writing some pytester calls a little nicer, such as when
creating a directory based on a `.joinpath()` call. We previously needed
to cast that intermediate value to a `str`.
As discussed in https://github.com/pytest-dev/pytest/pull/9800, this uses
monkeypatch to change directories for two tests in `test_collection.py`, to preserve
their original purpose (even if just removing it doesn't seem to affect the outcome).
Getting the current working directory has no side-effects, so these context managers were no-ops.
Discovered because Path.__enter__ is deprecated in Python 3.11, for removal in 3.13.
This will trigger a deploy directly when a tag is pushed, avoiding
having to wait for another lenghty build.
This is safe since we only push release tags after the release PR has passed all its tests.
See comments in #9793.
The left/right operands produced when `verbose > 1` should not contain newlines, because they are used to
build the `summary` string. The `assertrepr_compare` function returns a list of lines, and the summary is one of those lines and should not contain newlines itself.
Fix#9742
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Bruno Oliveira <nicoddemus@gmail.com>
The change from `path not in confuctdir.parents` to the `relative_to`
check in 0c98f19231 broke picking up
conftest files when running against an installed package/site-packages.
See the issue for more details.
Fix#9767.
Including the file name is enough to let the user know what the problem is.
The same is not needed for `.ini` files because the error message includes the path to the file by default.
Fix#9730
It is tempting to use `monkeypatch` to replace the other mechanisms in pytester which change global
state: `CwdSnapshot`, `SysModulesSnapshot`, `SysPathsSnapshot`, however those are more delicate
than they look at first glance so leaving those alone for now.
Close#9708
The dummy modules we introduce in `insert_missing_modules` (due to #7856 and #7859)
would cause problems if the dummy modules actually end up replacing modules
which could be imported normally because they are available in `PYTHONPATH`.
Now we attempt to first import the module via normal mechanisms, and only
introduce the dummy modules if the intermediary modules don't actually exist.
Close#9645
In the following
@pytest.mark.parametrize(..., ids=[val])
the ID values are only allowed to be `str`, `float`, `int` or `bool`.
In the following
@pytest.mark.parametrize(..., [val])
@pytest.mark.parametrize(..., [pytest.param(..., id=val])
a different code path is used, which also allows `bytes`, `complex`,
`re.Pattern`, `Enum` and anything with a `__name__`.
In the interest of consistency, use the latter code path for all cases.
originally reproduced with this pollution set:
```
testing/test_assertrewrite.py::TestEarlyRewriteBailout::test_pattern_contains_subdirectories
testing/test_assertrewrite.py::TestRewriteOnImport::test_remember_rewritten_modules
```
This fixes#9610.
pytest 7.0.0 (unintentionally) changed `UnitTestFunction.obj`'s' behavior
to match `Function.obj`. That is probably a good thing to have, however
it evidently causes some regressions as described in the issue, so
restore the previous behavior for now. In the future we might want to
make this change again, but with proper consideration.
re: review from @asottile that this should only get imported in the function
modify the else/if logic since inside the function we already know the python version is >= 3.10, and just have to know if it is 3.11 or greater
I still think it makes sense to always serve docs from the release branch (rather than the latest tagged release), but that means we need to update it on new releases.
The docs already assume an 'upstream' remote, so we can only fetch from there instead of fetching all remotes. We also don't need to hardcode the remote URL.
alinsa_vix in Discord noticed that we are not testing Python 3.9 and 3.10 on macOS, which seems strange. Maybe this is due how to macOS CI resources were quite scarce for a while, but I believe this has improved since.
* releasing: Add template for major releases
With pytest 6.0.0, we still used a manual releasing workflow (at least if I
remember correctly), and apparently we never wrote a release announcement
template for major releases. Instead, the minor release template claimed that
the release would contain "breaking changes", which doesn't seem reasonable.
Thus, this adds a new major template based on the former minor template, and
adjusts the latter to only mention fixes and new features instead.
* Update scripts/release.minor.rst
Co-authored-by: Bruno Oliveira <nicoddemus@gmail.com>
Co-authored-by: Bruno Oliveira <nicoddemus@gmail.com>
* Add additional docs for uncooperative ctor deprecation
Fixes#9488
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Break up long line
* Recommend kwonly args
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit only refactors, it does not change or add functionality yet. Public
API is retained. Reason or refactoring:
User provided parameter IDs (e.g. Metafunc.parametrize(ids=...)) had so far
only been used to calculate a unique test ID for each test invocation. That
test ID was a joined string where each parameter contributed some partial ID.
We're soon going to reuse functionality to generate parameter keys for
reorder_items and FixtureDef cache. We will be interested in the partial
IDs, and only if they originate from explicit user information. Refactoring
makes logic and data accessible for reuse, and increases cohesion in general.
The docstring (and function name itself) described things as if IDs are
being assigned to the argnames, but actually they're assigned to the
parameter sets.
If we do the `update`s in the right order, we can avoid the `mark.name
not in self.keywords` check, since `self.keywords` starts out clean and
`update` will override previously set keywords.
By my analysis, this deleted code block has no effect:
1. `self.keywords` is `update`d with `callspec.marks`.
2. `self.own_markers` is `update`d with `callspec.marks`.
3. `self.keywords` is `update`d with `self.own_markers`.
So together steps 2+3 completely undo step 1.
This does have a slight semantic change: in a node hierarchy parent ->
child, if parent has a marker applied, then child is constructed, then
`parent.themarker = "overridden"`, previously
`child.keywords['themarker']` would return `True`, now it returns
`"overridden"`. But that's actually what I would have expected so I see
it as more of a bugfix.
(except `Instance`)
Currently, `Function` does this manually, but other node types don't get
their markers added to their `keywords`, but they should, if only for
consistency.
Marks are added to keywords in three places:
- `Node.add_marker`: name -> `Mark`
- `Function.__init__(callspec)`: name -> `Mark`
- `Function.__init__ iter_markers`: name -> True
I think it should be consistent, which will also help with some upcoming
code cleaning. The `Mark` seems more useful than just a `True`, so
switch to that.
* Add docs on pytest.warns(None) deprecation
* Add new section for common warnings use cases
* Fix references for warnings use cases
* Fix reference link
I think this named function makes the code a bit easier to understand.
Also change the check to explicitly check for "is a sub-path of" instead
of the previous check which only worked assuming that path is within
confcutdir or a direct parent of it.
This made the cache not work as intended, causing a major slowdown.
See #9478 for discussion and context.
Authored-by: Anthony Sottile <asottile@umich.edu>
Before 7.0.0rc1, the function accepted `Union[str, py.path.local]`, and
`py.path.local` compares equal to the string path, so a user was able to
pass the path as a string and it would work. In 7.0.0rc1 we changed the
`py.path.local` to `Path` which doesn't compare equal to the string
path, which breaks compatibility (e.g. the `sybil` package).
This restores compatibility for this function by accepting any
`os.PathLike[str]` and only comparing the string representations.
Since commit 89f0b5b5a2 cases as in the
added test started to fail, like they do for the standard pytest names
(`setup_module` etc). But the name `setup` in particular is way too
common for us to start taking it over more aggressively, so restore the
previous behavior which required the object to be callable.
Fix#9391.
To backport a PR, e.g. 1000, to another branch, e.g. `7.0.x`, add a
label `backport 7.0.x` to the PR. This will trigger a workflow which
will create a branch `backport-1000-to-7.0.x` based on the `7.0.x`
branch with a cherry-pick of the PR's merge commit, and create a new PR
for it against the `7.0.x` branch.
It is very simplistic, for instance it doesn't handle cherry-pick
failure gracefully, doesn't validate the state of the PR, doesn't check
if the branch already exists, etc. But we can improve on it later as
needed.
Finally, PRs created by github actions do not themselves trigger further
actions, i.e. the PR isn't checked. You need to close & reopen the PR
for the checks to trigger. There are workarounds for this but they are
either less secure or require more setup.
The `pytest_pycollector_makeitem` argument `collector` is currently
annotated with type `PyCollector`. As part of #7469, that would have
required us to expose it in the public API. But really it's an
implementation detail, not something we want to expose. So replace the
annotation with the concrete python collector types that are passed.
Strictly speaking, `pytest_pycollector_makeitem` is called from
`PyCollector.collect()`, so the new type annotation is incorrect if
another type subclasses `PyCollector`. But the set of python collectors
is closed (mapping to language constructs), and the type is private, so
there shouldn't be any other deriving classes, and we can consider it
effectively sealed (unfortunately Python does not provide a way to
express this - yet?).
* scripts: Use release branch for changelog URL
With a prerelease, /stable won't show the correct changelog.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
stale-issue-message:"This issue is stale because it has the `status: needs information` label and requested follow-up information was not provided for 14 days."
close-issue-message:"This issue was closed because it has the `status: needs information` label and follow-up information has not been provided for 7 days since being marked as stale."
Contributions are highly welcomed and appreciated. Every little bit of help counts,
so do not hesitate!
..contents::
:depth:2
:backlinks:none
.._submitfeedback:
@@ -50,6 +46,8 @@ Fix bugs
--------
Look through the `GitHub issues for bugs <https://github.com/pytest-dev/pytest/labels/type:%20bug>`_.
See also the `"good first issue" issues <https://github.com/pytest-dev/pytest/labels/good%20first%20issue>`_
that are friendly to new contributors.
:ref:`Talk <contact>` to developers to find out how you can fix specific bugs. To indicate that you are going
to work on a particular issue, add a comment to that effect on the specific issue.
@@ -195,11 +193,12 @@ Short version
~~~~~~~~~~~~~
#. Fork the repository.
#. Fetch tags from upstream if necessary (if you cloned only main `git fetch --tags https://github.com/pytest-dev/pytest`).
#. Enable and install `pre-commit <https://pre-commit.com>`_ to ensure style-guides and code checks are followed.
#. Follow **PEP-8** for naming and `black <https://github.com/psf/black>`_ for formatting.
#. Follow `PEP-8<https://www.python.org/dev/peps/pep-0008/>`_ for naming.
#. Tests are run using ``tox``::
tox -e linting,py37
tox -e linting,py39
The test environments above are usually enough to cover most cases locally.
@@ -221,7 +220,7 @@ changes you want to review and merge. Pull requests are stored on
Once you send a pull request, we can discuss its potential modifications and
even add more commits to it later on. There's an excellent tutorial on how Pull
Requests work in the
`GitHub Help Center <https://help.github.com/articles/using-pull-requests/>`_.
`GitHub Help Center <https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests>`_.
Here is a simple overview, with pytest-specific bits:
@@ -234,6 +233,7 @@ Here is a simple overview, with pytest-specific bits:
@@ -242,6 +242,11 @@ Here is a simple overview, with pytest-specific bits:
be released in micro releases whereas features will be released in
minor releases and incompatible changes in major releases.
You will need the tags to test locally, so be sure you have the tags from the main repository. If you suspect you don't, set the main repository as upstream and fetch the tags::
@@ -265,35 +270,35 @@ Here is a simple overview, with pytest-specific bits:
#. Run all the tests
You need to have Python 3.7 available in your system. Now
You need to have Python 3.8 or later available in your system. Now
running tests is as simple as issuing this command::
$ tox -e linting,py37
$ tox -e linting,py39
This command will run tests via the "tox" tool against Python 3.7
This command will run tests via the "tox" tool against Python 3.9
and also perform "lint" coding-style checks.
#. You can now edit your local working copy and run the tests again as necessary. Please follow PEP-8 for naming.
#. You can now edit your local working copy and run the tests again as necessary. Please follow `PEP-8<https://www.python.org/dev/peps/pep-0008/>`_ for naming.
You can pass different options to ``tox``. For example, to run tests on Python 3.7 and pass options to pytest
You can pass different options to ``tox``. For example, to run tests on Python 3.9 and pass options to pytest
(e.g. enter pdb on failure) to pytest you can do::
$ tox -e py37 -- --pdb
$ tox -e py39 -- --pdb
Or to only run tests in a particular test module on Python 3.7::
Or to only run tests in a particular test module on Python 3.9::
$ tox -e py37 -- testing/test_config.py
$ tox -e py39 -- testing/test_config.py
When committing, ``pre-commit`` will re-format the files if necessary.
#. If instead of using ``tox`` you prefer to run the tests directly, then we suggest to create a virtual environment and use
an editable install with the ``testing`` extra::
an editable install with the ``dev`` extra::
$ python3 -m venv .venv
$ source .venv/bin/activate # Linux
$ .venv/Scripts/activate.bat # Windows
$ pip install -e ".[testing]"
$ pip install -e ".[dev]"
Afterwards, you can edit the files and run pytest normally::
@@ -378,7 +383,7 @@ them.
Backporting bug fixes for the next patch release
------------------------------------------------
Pytest makes feature release every few weeks or months. In between, patch releases
Pytest makes a feature release every few weeks or months. In between, patch releases
are made to the previous feature release, containing bug fixes only. The bug fixes
usually fix regressions, but may be any change that should reach users before the
next feature release.
@@ -387,10 +392,17 @@ Suppose for example that the latest release was 1.2.3, and you want to include
a bug fix in 1.2.4 (check https://github.com/pytest-dev/pytest/releases for the
actual latest release). The procedure for this is:
#. First, make sure the bug is fixed the ``main`` branch, with a regular pull
#. First, make sure the bug is fixed in the ``main`` branch, with a regular pull
request, as described above. An exception to this is if the bug fix is not
applicable to ``main`` anymore.
Automatic method:
Add a ``backport 1.2.x`` label to the PR you want to backport. This will create
a backport PR against the ``1.2.x`` branch.
Manual method:
#.``git checkout origin/1.2.x -b backport-XXXX`` # use the main PR number here
#. Locate the merge commit on the PR, in the *merged* message, for example:
#. For major and minor releases, change the default version in the `Read the Docs Settings <https://readthedocs.org/dashboard/pytest/advanced/>`_ to the new branch.
#. Send an email announcement with the contents from::
@@ -14,7 +14,7 @@ Each file should be named like ``<ISSUE>.<TYPE>.rst``, where
``<ISSUE>`` is an issue number, and ``<TYPE>`` is one of:
*``feature``: new user facing features, like new command-line options and new behavior.
*``improvement``: improvement of existing functionality, usually without requiring user intervention (for example, new fields being written in ``--junitxml``, improved colors in terminal, etc).
*``improvement``: improvement of existing functionality, usually without requiring user intervention (for example, new fields being written in ``--junit-xml``, improved colors in terminal, etc).
*``bugfix``: fixes a bug.
*``doc``: documentation improvement, like rewording an entire session or adding missing docs.
There can be many different definitions of "success". Pytest can run many nose_ and unittest_ tests by default, so using pytest as your testrunner may be possible from day 1. Job done, right?
There can be many different definitions of "success". Pytest can run many unittest_ tests by default, so using pytest as your testrunner may be possible from day 1. Job done, right?
It may be after the month is up, the partner project decides that pytest is not right for it. That's okay - hopefully the pytest team will also learn something about its weaknesses or deficiencies.
@@ -22,7 +22,7 @@ b) transitional: the old and new API don't conflict
We will only start the removal of deprecated functionality in major releases (e.g. if we deprecate something in 3.0 we will start to remove it in 4.0), and keep it around for at least two minor releases (e.g. if we deprecate something in 3.9 and 4.0 is the next release, we start to remove it in 5.0, not in 4.0).
A deprecated feature scheduled to be removed in major version X will use the warning class `PytestRemovedInXWarning` (a subclass of :class:`~pytest.PytestDeprecationwarning`).
A deprecated feature scheduled to be removed in major version X will use the warning class `PytestRemovedInXWarning` (a subclass of :class:`~pytest.PytestDeprecationWarning`).
When the deprecation expires (e.g. 4.0 is released), we won't remove the deprecated functionality immediately, but will use the standard warning filters to turn `PytestRemovedInXWarning` (e.g. `PytestRemovedIn4Warning`) into **errors** by default. This approach makes it explicit that removal is imminent, and still gives you time to turn the deprecated feature into a warning instead of an error so it can be dealt with in your own time. In the next minor release (e.g. 4.1), the feature will be effectively removed.
@@ -77,3 +77,21 @@ Deprecation Roadmap
Features currently deprecated and removed in previous releases can be found in :ref:`deprecations`.
We track future deprecation and removal of features using milestones and the `deprecation <https://github.com/pytest-dev/pytest/issues?q=label%3A%22type%3A+deprecation%22>`_ and `removal <https://github.com/pytest-dev/pytest/labels/type%3A%20removal>`_ labels on GitHub.
Python version support
======================
Released pytest versions support all Python versions that are actively maintained at the time of the release:
============== ===================
pytest version min. Python version
============== ===================
8.0+ 3.8+
7.1+ 3.7+
6.2 - 7.0 3.6+
5.0 - 6.1 3.5+
3.3 - 4.6 2.7, 3.4+
============== ===================
`Status of Python Versions <https://devguide.python.org/versions/>`__.
The ``pytest.Instance`` collector type has been removed.
..deprecated:: 8.2
Previously, Python test methods were collected as :class:`~pytest.Class` -> ``Instance`` -> :class:`~pytest.Function`.
Now :class:`~pytest.Class` collects the test methods directly.
Traditionally :func:`pytest.importorskip` will capture :class:`ImportError`, with the original intent being to skip
tests where a dependent module is not installed, for example testing with different dependencies.
Most plugins which reference ``Instance`` do so in order to ignore or skip it,
using a check such as ``if isinstance(node, Instance): return``.
Such plugins should simply remove consideration of ``Instance`` on pytest>=7.
However, to keep such uses working, a dummy type has been instanted in ``pytest.Instance`` and ``_pytest.python.Instance``,
and importing it emits a deprecation warning. This will be removed in pytest 8.
However some packages might be installed in the system, but are not importable due to
some other issue, for example, a compilation error or a broken installation. In those cases :func:`pytest.importorskip`
would still silently skip the test, but more often than not users would like to see the unexpected
error so the underlying issue can be fixed.
In ``8.2`` the ``exc_type`` parameter has been added, giving users the ability of passing :class:`ModuleNotFoundError`
to skip tests only if the module cannot really be found, and not because of some other error.
Catching only :class:`ModuleNotFoundError` by default (and letting other errors propagate) would be the best solution,
however for backward compatibility, pytest will keep the existing behavior but raise an warning if:
1. The captured exception is of type :class:`ImportError`, and:
2. The user does not pass ``exc_type`` explicitly.
If the import attempt raises :class:`ModuleNotFoundError` (the usual case), then the module is skipped and no
warning is emitted.
This way, the usual cases will keep working the same way, while unexpected errors will now issue a warning, with
users being able to supress the warning by passing ``exc_type=ImportError`` explicitly.
In ``9.0``, the warning will turn into an error, and in ``9.1``:func:`pytest.importorskip` will only capture
:class:`ModuleNotFoundError` by default and no warnings will be issued anymore -- but users can still capture
:class:`ImportError` by passing it to ``exc_type``.
.._node-ctor-fspath-deprecation:
@@ -56,6 +69,10 @@ Plugins which implement custom items and collectors are encouraged to replace
``fspath`` parameters (``py.path.local``) with ``path`` parameters
(``pathlib.Path``), and drop any other usage of the ``py`` library if possible.
If possible, plugins with custom items should use :ref:`cooperative
constructors <uncooperative-constructors-deprecated>` to avoid hardcoding
arguments they only pass on to the superclass.
..note::
The name of the :class:`~_pytest.nodes.Node` arguments and attributes (the
new attribute being ``path``) is **the opposite** of the situation for
@@ -66,12 +83,54 @@ Plugins which implement custom items and collectors are encouraged to replace
resolved in future versions as we slowly get rid of the :pypi:`py`
dependency (see :issue:`9283` for a longer discussion).
Due to the ongoing migration of methods like :meth:`~_pytest.Item.reportinfo`
Due to the ongoing migration of methods like :meth:`~pytest.Item.reportinfo`
which still is expected to return a ``py.path.local`` object, nodes still have
both ``fspath`` (``py.path.local``) and ``path`` (``pathlib.Path``) attributes,
no matter what argument was used in the constructor. We expect to deprecate the
``fspath`` attribute in a future release.
Configuring hook specs/impls using markers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Before pluggy, pytest's plugin library, was its own package and had a clear API,
pytest just used ``pytest.mark`` to configure hooks.
The :py:func:`pytest.hookimpl` and :py:func:`pytest.hookspec` decorators
have been available since years and should be used instead.
..code-block::python
@pytest.mark.tryfirst
defpytest_runtest_call():...
# or
defpytest_runtest_call():...
pytest_runtest_call.tryfirst=True
should be changed to:
..code-block::python
@pytest.hookimpl(tryfirst=True)
defpytest_runtest_call():...
Changed ``hookimpl`` attributes:
*``tryfirst``
*``trylast``
*``optionalhook``
*``hookwrapper``
Changed ``hookwrapper`` attributes:
*``firstresult``
*``historic``
.._legacy-path-hooks-deprecated:
``py.path.local`` arguments for hooks replaced with ``pathlib.Path``
@@ -81,11 +140,11 @@ no matter what argument was used in the constructor. We expect to deprecate the
In order to support the transition from ``py.path.local`` to :mod:`pathlib`, the following hooks now receive additional arguments:
*:func:`pytest_ignore_collect(collection_path: pathlib.Path) <_pytest.hookspec.pytest_ignore_collect>` as equivalent to ``path``
*:func:`pytest_collect_file(file_path: pathlib.Path) <_pytest.hookspec.pytest_collect_file>` as equivalent to ``path``
*:func:`pytest_pycollect_makemodule(module_path: pathlib.Path) <_pytest.hookspec.pytest_pycollect_makemodule>` as equivalent to ``path``
*:func:`pytest_report_header(start_path: pathlib.Path) <_pytest.hookspec.pytest_report_header>` as equivalent to ``startdir``
*:func:`pytest_report_collectionfinish(start_path: pathlib.Path) <_pytest.hookspec.pytest_report_collectionfinish>` as equivalent to ``startdir``
*:hook:`pytest_ignore_collect(collection_path: pathlib.Path) <pytest_ignore_collect>` as equivalent to ``path``
*:hook:`pytest_collect_file(file_path: pathlib.Path) <pytest_collect_file>` as equivalent to ``path``
*:hook:`pytest_pycollect_makemodule(module_path: pathlib.Path) <pytest_pycollect_makemodule>` as equivalent to ``path``
*:hook:`pytest_report_header(start_path: pathlib.Path) <pytest_report_header>` as equivalent to ``startdir``
*:hook:`pytest_report_collectionfinish(start_path: pathlib.Path) <pytest_report_collectionfinish>` as equivalent to ``startdir``
The accompanying ``py.path.local`` based paths have been deprecated: plugins which manually invoke those hooks should only pass the new ``pathlib.Path`` arguments, and users should change their hook implementations to use the new ``pathlib.Path`` arguments.
@@ -118,16 +177,248 @@ Directly constructing the following classes is now deprecated:
These constructors have always been considered private, but now issue a deprecation warning, which may become a hard error in pytest 8.
.._cmdline-preparse-deprecated:
.._diamond-inheritance-deprecated:
Diamond inheritance between :class:`pytest.Collector` and :class:`pytest.Item`
Defining a custom pytest node type which is both an :class:`~pytest.Item` and a :class:`~pytest.Collector` (e.g. :class:`~pytest.File`) now issues a warning.
It was never sanely supported and triggers hard to debug errors.
Some plugins providing linting/code analysis have been using this as a hack.
Instead, a separate collector node should be used, which collects the item. See
:ref:`non-python tests` for an example, as well as an `example pr fixing inheritance`_.
If custom subclasses of nodes like :class:`pytest.Item` override the
``__init__`` method, they should take ``**kwargs``. Thus,
..code-block::python
classCustomItem(pytest.Item):
def__init__(self,name,parent,additional_arg):
super().__init__(name,parent)
self.additional_arg=additional_arg
should be turned into:
..code-block::python
classCustomItem(pytest.Item):
def__init__(self,*,additional_arg,**kwargs):
super().__init__(**kwargs)
self.additional_arg=additional_arg
to avoid hard-coding the arguments pytest can pass to the superclass.
See :ref:`non-python tests` for a full example.
For cases without conflicts, no deprecation warning is emitted. For cases with
conflicts (such as :class:`pytest.File` now taking ``path`` instead of
``fspath``, as :ref:`outlined above <node-ctor-fspath-deprecation>`), a
deprecation warning is now raised.
Applying a mark to a fixture function
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
..deprecated:: 7.4
Applying a mark to a fixture function never had any effect, but it is a common user error.
..code-block::python
@pytest.mark.usefixtures("clean_database")
@pytest.fixture
defuser()->User:...
Users expected in this case that the ``usefixtures`` mark would have its intended effect of using the ``clean_database`` fixture when ``user`` was invoked, when in fact it has no effect at all.
Now pytest will issue a warning when it encounters this problem, and will raise an error in the future versions.
Returning non-None value in test functions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
..deprecated:: 7.2
A :class:`pytest.PytestReturnNotNoneWarning` is now emitted if a test function returns something other than `None`.
This prevents a common mistake among beginners that expect that returning a `bool` would cause a test to pass or fail, for example:
..code-block::python
@pytest.mark.parametrize(
["a","b","result"],
[
[1,2,5],
[2,3,8],
[5,3,18],
],
)
deftest_foo(a,b,result):
returnfoo(a,b)==result
Given that pytest ignores the return value, this might be surprising that it will never fail.
The proper fix is to change the `return` to an `assert`:
..code-block::python
@pytest.mark.parametrize(
["a","b","result"],
[
[1,2,5],
[2,3,8],
[5,3,18],
],
)
deftest_foo(a,b,result):
assertfoo(a,b)==result
The ``yield_fixture`` function/decorator
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
..deprecated:: 6.2
``pytest.yield_fixture`` is a deprecated alias for :func:`pytest.fixture`.
It has been so for a very long time, so can be search/replaced safely.
Removed Features and Breaking Changes
-------------------------------------
As stated in our :ref:`backwards-compatibility` policy, deprecated features are removed only in major releases after
an appropriate period of deprecation has passed.
Some breaking changes which could not be deprecated are also listed.
.._nose-deprecation:
Support for tests written for nose
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
..deprecated:: 7.2
..versionremoved:: 8.0
Support for running tests written for `nose <https://nose.readthedocs.io/en/latest/>`__ is now deprecated.
``nose`` has been in maintenance mode-only for years, and maintaining the plugin is not trivial as it spills
over the code base (see :issue:`9886` for more details).
setup/teardown
^^^^^^^^^^^^^^
One thing that might catch users by surprise is that plain ``setup`` and ``teardown`` methods are not pytest native,
they are in fact part of the ``nose`` support.
..code-block::python
classTest:
defsetup(self):
self.resource=make_resource()
defteardown(self):
self.resource.close()
deftest_foo(self):...
deftest_bar(self):...
Native pytest support uses ``setup_method`` and ``teardown_method`` (see :ref:`xunit-method-setup`), so the above should be changed to:
..code-block::python
classTest:
defsetup_method(self):
self.resource=make_resource()
defteardown_method(self):
self.resource.close()
deftest_foo(self):...
deftest_bar(self):...
This is easy to do in an entire code base by doing a simple find/replace.
@with_setup
^^^^^^^^^^^
Code using `@with_setup <with-setup-nose>`_ such as this:
Defining a custom pytest node type which is both an :class:`pytest.Item <Item>` and a :class:`pytest.Collector <Collector>` (e.g. :class:`pytest.File <File>`) now issues a warning.
It was never sanely supported and triggers hard to debug errors.
Some plugins providing linting/code analysis have been using this as a hack.
Instead, a separate collector node should be used, which collects the item. See
:ref:`non-python tests` for an example, as well as an `example pr fixing inheritance`_.
Running `pytest pkg/__init__.py` now collects the `pkg/__init__.py` file (module) only.
Previously, it collected the entire `pkg` package, including other test files in the directory, but excluding tests in the `__init__.py` file itself
(unless :confval:`python_files` was changed to allow `__init__.py` file).
To collect the entire package, specify just the directory: `pytest pkg`.
The ``pytest.collect`` module
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
..deprecated:: 6.0
..versionremoved:: 7.0
The ``pytest.collect`` module is no longer part of the public API, all its names
should now be imported from ``pytest`` directly instead.
It has been so for a very long time, so can be search/replaced safely.
The ``pytest_warning_captured`` hook
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
..deprecated:: 6.0
..versionremoved:: 7.0
This hook has an `item` parameter which cannot be serialized by ``pytest-xdist``.
Use the ``pytest_warning_recored`` hook instead, which replaces the ``item`` parameter
Use the ``pytest_warning_recorded`` hook instead, which replaces the ``item`` parameter
by a ``nodeid`` parameter.
The ``pytest.collect`` module
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
..deprecated:: 6.0
The ``pytest.collect`` module is no longer part of the public API, all its names
should now be imported from ``pytest`` directly instead.
The ``pytest._fillfuncargs`` function
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
..deprecated:: 6.0
..versionremoved:: 7.0
This function was kept for backward compatibility with an older plugin.
@@ -281,12 +676,6 @@ it, use `function._request._fillfixtures()` instead, though note this is not
a public API and may break in the future.
Removed Features
----------------
As stated in our :ref:`backwards-compatibility` policy, deprecated features are removed only in major releases after
an appropriate period of deprecation has passed.
``--no-print-logs`` command-line option
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -324,8 +713,8 @@ at some point, depending on the plans for the plugins and number of users using
..versionremoved:: 6.0
The ``pytest_collect_directory`` has not worked properly for years (it was called
but the results were ignored). Users may consider using :func:`pytest_collection_modifyitems <_pytest.hookspec.pytest_collection_modifyitems>` instead.
The ``pytest_collect_directory`` hook has not worked properly for years (it was called
but the results were ignored). Users may consider using :hook:`pytest_collection_modifyitems` instead.
TerminalReporter.writer
~~~~~~~~~~~~~~~~~~~~~~~
@@ -372,7 +761,7 @@ By using ``legacy`` you will keep using the legacy/xunit1 format when upgrading
pytest 6.0, where the default format will be ``xunit2``.
In order to let users know about the transition, pytest will issue a warning in case
the ``--junitxml`` option is given in the command line but ``junit_family`` is not explicitly
the ``--junit-xml`` option is given in the command line but ``junit_family`` is not explicitly
configured in ``pytest.ini``.
Services known to support the ``xunit2`` format:
@@ -549,8 +938,7 @@ Applying marks to values of a ``pytest.mark.parametrize`` call is now deprecated
(50,500),
],
)
deftest_foo(a,b):
...
deftest_foo(a,b):...
This code applies the ``pytest.mark.xfail(reason="flaky")`` mark to the ``(6, 36)`` value of the above parametrization
call.
@@ -573,8 +961,7 @@ To update the code, use ``pytest.param``:
(50,500),
],
)
deftest_foo(a,b):
...
deftest_foo(a,b):...
.._pytest_funcarg__ prefix deprecated:
@@ -725,15 +1112,13 @@ This is just a matter of renaming the fixture as the API is the same:
..code-block::python
deftest_foo(record_xml_property):
...
deftest_foo(record_xml_property):...
Change to:
..code-block::python
deftest_foo(record_property):
...
deftest_foo(record_property):...
.._passing command-line string to pytest.main deprecated:
@@ -828,7 +1213,7 @@ that are then turned into proper test methods. Example:
..code-block::python
defcheck(x,y):
assertx**x==y
assertx**x==y
deftest_squared():
@@ -843,7 +1228,7 @@ This form of test function doesn't support fixtures properly, and users should s
@pytest.mark.parametrize("x, y",[(2,4),(3,9)])
deftest_squared(x,y):
assertx**x==y
assertx**x==y
.._internal classes accessed through node deprecated:
@@ -246,9 +246,9 @@ You can ask which markers exist for your test suite - the list includes our just
@pytest.mark.usefixtures(fixturename1, fixturename2, ...): mark tests as needing all of the specified fixtures. see https://docs.pytest.org/en/stable/explanation/fixtures.html#usefixtures
@pytest.mark.tryfirst: mark a hook implementation function such that the plugin machinery will try to call it first/as early as possible.
@pytest.mark.tryfirst: mark a hook implementation function such that the plugin machinery will try to call it first/as early as possible. DEPRECATED, use @pytest.hookimpl(tryfirst=True) instead.
@pytest.mark.trylast: mark a hook implementation function such that the plugin machinery will try to call it last/as late as possible.
@pytest.mark.trylast: mark a hook implementation function such that the plugin machinery will try to call it last/as late as possible. DEPRECATED, use @pytest.hookimpl(trylast=True) instead.
For an example on how to add and work with markers from a plugin, see
@@ -346,7 +346,7 @@ Custom marker and command line option to control test runs
Plugins can provide custom markers and implement specific behaviour
based on it. This is a self-contained example which adds a command
line option and a parametrized test function marker to run tests
specifies via named environments:
specified via named environments:
..code-block::python
@@ -375,7 +375,7 @@ specifies via named environments:
pytest.skip("test requires env in {!r}".format(envnames))
pytest.skip(f"test requires env in {envnames!r}")
A test file using this local plugin:
@@ -397,7 +397,7 @@ the test needs:
$ pytest -E stage2
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-7.x.y, pluggy-1.x.y
platform linux -- Python 3.x.y, pytest-8.x.y, pluggy-1.x.y
rootdir: /home/sweet/project
collected 1 item
@@ -411,7 +411,7 @@ and here is one that specifies exactly the environment needed:
$ pytest -E stage1
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-7.x.y, pluggy-1.x.y
platform linux -- Python 3.x.y, pytest-8.x.y, pluggy-1.x.y
rootdir: /home/sweet/project
collected 1 item
@@ -438,9 +438,9 @@ The ``--markers`` option always gives you a list of available markers:
@pytest.mark.usefixtures(fixturename1, fixturename2, ...): mark tests as needing all of the specified fixtures. see https://docs.pytest.org/en/stable/explanation/fixtures.html#usefixtures
@pytest.mark.tryfirst: mark a hook implementation function such that the plugin machinery will try to call it first/as early as possible.
@pytest.mark.tryfirst: mark a hook implementation function such that the plugin machinery will try to call it first/as early as possible. DEPRECATED, use @pytest.hookimpl(tryfirst=True) instead.
@pytest.mark.trylast: mark a hook implementation function such that the plugin machinery will try to call it last/as late as possible.
@pytest.mark.trylast: mark a hook implementation function such that the plugin machinery will try to call it last/as late as possible. DEPRECATED, use @pytest.hookimpl(trylast=True) instead.
.._`passing callables to custom markers`:
@@ -528,7 +528,7 @@ test function. From a conftest file we can read it like this:
Here is an example ``conftest.py`` (extracted from Ali Afshar's special purpose `pytest-yamlwsgi`_ plugin). This ``conftest.py`` will collect ``test*.yaml`` files and will execute the yaml-formatted content as custom tests:
@@ -28,7 +28,7 @@ now execute the test specification:
nonpython $ pytest test_simple.yaml
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-7.x.y, pluggy-1.x.y
platform linux -- Python 3.x.y, pytest-8.x.y, pluggy-1.x.y
rootdir: /home/sweet/project/nonpython
collected 2 items
@@ -64,7 +64,7 @@ consulted when reporting in ``verbose`` mode:
nonpython $ pytest -v
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-7.x.y, pluggy-1.x.y -- $PYTHON_PREFIX/bin/python
platform linux -- Python 3.x.y, pytest-8.x.y, pluggy-1.x.y -- $PYTHON_PREFIX/bin/python
cachedir: .pytest_cache
rootdir: /home/sweet/project/nonpython
collecting ... collected 2 items
@@ -90,7 +90,7 @@ interesting to just look at the collection tree:
nonpython $ pytest --collect-only
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-7.x.y, pluggy-1.x.y
platform linux -- Python 3.x.y, pytest-8.x.y, pluggy-1.x.y
Broadly speaking, a flaky test indicates that the test relies on some system state that is not being appropriately controlled - the test environment is not sufficiently isolated. Higher level tests are more likely to be flaky as they rely on more state.
Flaky tests sometimes appear when a test suite is run in parallel (such as use of pytest-xdist). This can indicate a test is reliant on test ordering.
Flaky tests sometimes appear when a test suite is run in parallel (such as use of `pytest-xdist`_). This can indicate a test is reliant on test ordering.
- Perhaps a different test is failing to clean up after itself and leaving behind data which causes the flaky test to fail.
- The flaky test is reliant on data from a previous test that doesn't clean up after itself, and in parallel runs that previous test is not always present
@@ -30,9 +30,22 @@ Overly strict assertion
Overly strict assertions can cause problems with floating point comparison as well as timing issues. :func:`pytest.approx` is useful here.
Thread safety
~~~~~~~~~~~~~
Pytest features
^^^^^^^^^^^^^^^
pytest is single-threaded, executing its tests always in the same thread, sequentially, never spawning any threads itself.
Even in case of plugins which run tests in parallel, for example `pytest-xdist`_, usually work by spawning multiple *processes* and running tests in batches, without using multiple threads.
It is of course possible (and common) for tests and fixtures to spawn threads themselves as part of their testing workflow (for example, a fixture that starts a server thread in the background, or a test which executes production code that spawns threads), but some care must be taken:
* Make sure to eventually wait on any spawned threads -- for example at the end of a test, or during the teardown of a fixture.
* Avoid using primitives provided by pytest (:func:`pytest.warns`, :func:`pytest.raises`, etc) from multiple threads, as they are not thread-safe.
If your test suite uses threads and your are seeing flaky test results, do not discount the possibility that the test is implicitly using global state in pytest itself.
Related features
^^^^^^^^^^^^^^^^
Xfail strict
~~~~~~~~~~~~
@@ -52,10 +65,9 @@ Plugins
Rerunning any failed tests can mitigate the negative effects of flaky tests by giving them additional chances to pass, so that the overall build does not fail. Several pytest plugins support this:
*`flaky <https://github.com/box/flaky>`_
*`pytest-flakefinder <https://github.com/dropbox/pytest-flakefinder>`_ - `blog post <https://blogs.dropbox.com/tech/2016/03/open-sourcing-pytest-tools/>`_
*`pytest-replay <https://github.com/ESSS/pytest-replay>`_: This plugin helps to reproduce locally crashes or flaky tests observed during CI runs.
*`pytest-flakefinder <https://github.com/dropbox/pytest-flakefinder>`_ - `blog post <https://blogs.dropbox.com/tech/2016/03/open-sourcing-pytest-tools/>`_
Plugins to deliberately randomize tests can help expose tests with state problems:
@@ -94,7 +106,7 @@ Mark Lapierre discusses the `Pros and Cons of Quarantined Tests <https://dev.to/
CI tools that rerun on failure
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Azure Pipelines (the Azure cloud CI/CD tool, formerly Visual Studio Team Services or VSTS) has a feature to `identify flaky tests <https://docs.microsoft.com/en-us/azure/devops/release-notes/2017/dec-11-vsts#identify-flaky-tests>`_ and rerun failed tests.
Azure Pipelines (the Azure cloud CI/CD tool, formerly Visual Studio Team Services or VSTS) has a feature to `identify flaky tests <https://docs.microsoft.com/en-us/previous-versions/azure/devops/2017/dec-11-vsts?view=tfs-2017#identify-flaky-tests>`_ and rerun failed tests.
@@ -106,7 +118,7 @@ This is a limited list, please submit an issue or pull request to expand it!
* Gao, Zebao, Yalan Liang, Myra B. Cohen, Atif M. Memon, and Zhen Wang. "Making system user interactive tests repeatable: When and what should we control?." In *Software Engineering (ICSE), 2015 IEEE/ACM 37th IEEE International Conference on*, vol. 1, pp. 55-65. IEEE, 2015. `PDF <http://www.cs.umd.edu/~atif/pubs/gao-icse15.pdf>`__
* Palomba, Fabio, and Andy Zaidman. "Does refactoring of test smells induce fixing flaky tests?." In *Software Maintenance and Evolution (ICSME), 2017 IEEE International Conference on*, pp. 1-12. IEEE, 2017. `PDF in Google Drive <https://drive.google.com/file/d/10HdcCQiuQVgW3yYUJD-TSTq1NbYEprl0/view>`__
* Bell, Jonathan, Owolabi Legunsen, Michael Hilton, Lamyaa Eloussi, Tifany Yung, and Darko Marinov. "DeFlaker: Automatically detecting flaky tests." In *Proceedings of the 2018 International Conference on Software Engineering*. 2018. `PDF <https://www.jonbell.net/icse18-deflaker.pdf>`__
* Dutta, Saikat and Shi, August and Choudhary, Rutvik and Zhang, Zhekun and Jain, Aryaman and Misailovic, Sasa. "Detecting flaky tests in probabilistic and machine learning applications." In *Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA)*, pp. 211-224. ACM, 2020. `PDF <https://www.cs.cornell.edu/~saikatd/papers/flash-issta20.pdf>`__
Resources
^^^^^^^^^
@@ -124,3 +136,6 @@ Resources
* `Flaky Tests at Google and How We Mitigate Them <https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html>`_ by John Micco, 2016
*`Where do Google's flaky tests come from? <https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html>`_ by Jeff Listfield, 2017
@@ -12,41 +12,27 @@ For development, we recommend you use :mod:`venv` for virtual environments and
as well as the ``pytest`` package itself.
This ensures your code and dependencies are isolated from your system Python installation.
Next, place a ``pyproject.toml`` file in the root of your package:
Create a ``pyproject.toml`` file in the root of your repository as described in
:doc:`packaging:tutorials/packaging-projects`.
The first few lines should look like this:
..code-block::toml
[build-system]
requires=["setuptools>=42","wheel"]
build-backend="setuptools.build_meta"
requires=["hatchling"]
build-backend="hatchling.build"
and a ``setup.cfg`` file containing your package's metadata with the following minimum content:
[project]
name="PACKAGENAME"
version="PACKAGEVERSION"
..code-block::ini
[metadata]
name=PACKAGENAME
[options]
packages=find:
where ``PACKAGENAME`` is the name of your package.
..note::
If your pip version is older than ``21.3``, you'll also need a ``setup.py`` file:
..code-block::python
fromsetuptoolsimportsetup
setup()
where ``PACKAGENAME`` and ``PACKAGEVERSION`` are the name and version of your package respectively.
You can then install your package in "editable" mode by running from the same directory:
..code-block::bash
pip install -e .
pip install -e .
which lets you change your source code (both tests and application) and rerun tests at will.
@@ -65,8 +51,8 @@ Conventions for Python test discovery
* In those directories, search for ``test_*.py`` or ``*_test.py`` files, imported by their `test package name`_.
* From those files, collect test items:
*``test`` prefixed test functions or methods outside of class
*``test`` prefixed test functions or methods inside ``Test`` prefixed test classes (without an ``__init__`` method)
*``test`` prefixed test functions or methods outside of class.
*``test`` prefixed test functions or methods inside ``Test`` prefixed test classes (without an ``__init__`` method). Methods decorated with ``@staticmethod`` and ``@classmethods`` are also considered.
For examples of how to customize your test discovery :doc:`/example/pythoncollection`.
@@ -74,8 +60,10 @@ Within Python modules, ``pytest`` also discovers tests using the standard
If you do not use an editable install and use the ``src`` layout as above you need to extend the Python's
search path for module files to execute the tests against the local copy directly. You can do it in an
ad-hoc manner by setting the ``PYTHONPATH`` environment variable:
..code-block::bash
PYTHONPATH=src pytest
or in a permanent manner by using the :confval:`pythonpath` configuration variable and adding the
following to your ``pyproject.toml``:
..code-block::toml
[tool.pytest.ini_options]
pythonpath="src"
..note::
If you do not use an editable install and not use the ``src`` layout (``mypkg`` directly in the root
directory) you can rely on the fact that Python by default puts the current directory in ``sys.path`` to
import your package and run ``python -m pytest`` to execute the tests against the local copy directly.
See :ref:`pytest vs python -m pytest` for more information about the difference between calling ``pytest`` and
``python -m pytest``.
Note that this scheme has a drawback if you are using ``prepend``:ref:`import mode <import-modes>`
(which is the default): your test files must have **unique names**, because
``pytest`` will import them as *top-level* modules since there are no packages
to derive a full package name from. In other words, the test files in the example above will
be imported as ``test_app`` and ``test_view`` top-level modules by adding ``tests/`` to
``sys.path``.
If you need to have test modules with the same name, you might add ``__init__.py`` files to your
``tests`` folder and subfolders, changing them to packages:
..code-block::text
pyproject.toml
setup.cfg
mypkg/
...
tests/
__init__.py
foo/
__init__.py
test_view.py
bar/
__init__.py
test_view.py
Now pytest will load the modules as ``tests.foo.test_view`` and ``tests.bar.test_view``, allowing
you to have modules with the same name. But now this introduces a subtle problem: in order to load
the test modules from the ``tests`` directory, pytest prepends the root of the repository to
``sys.path``, which adds the side-effect that now ``mypkg`` is also importable.
This is problematic if you are using a tool like `tox`_ to test your package in a virtual environment,
because you want to test the *installed* version of your package, not the local code from the repository.
.._`src-layout`:
In this situation, it is **strongly** suggested to use a ``src`` layout where application root package resides in a
sub-directory of your root:
..code-block::text
pyproject.toml
setup.cfg
src/
mypkg/
__init__.py
app.py
view.py
tests/
__init__.py
foo/
__init__.py
test_view.py
bar/
__init__.py
test_view.py
This layout prevents a lot of common pitfalls and has many benefits, which are better explained in this excellent
`blog post by Ionel Cristian Mărieș <https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure>`_.
..note::
The new ``--import-mode=importlib`` (see :ref:`import-modes`) doesn't have
any of the drawbacks above because ``sys.path`` is not changed when importing
test modules, so users that run
into this issue are strongly encouraged to try it and report if the new option works well for them.
The ``src`` directory layout is still strongly recommended however.
Tests as part of application code
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -191,12 +152,11 @@ want to distribute them along with your application:
..code-block::text
pyproject.toml
setup.cfg
mypkg/
[src/]mypkg/
__init__.py
app.py
view.py
test/
tests/
__init__.py
test_app.py
test_view.py
@@ -254,6 +214,56 @@ Note that this layout also works in conjunction with the ``src`` layout mentione
much less surprising.
.._which-import-mode:
Choosing an import mode
^^^^^^^^^^^^^^^^^^^^^^^
For historical reasons, pytest defaults to the ``prepend``:ref:`import mode <import-modes>`
instead of the ``importlib`` import mode we recommend for new projects.
The reason lies in the way the ``prepend`` mode works:
Since there are no packages to derive a full package name from,
``pytest`` will import your test files as *top-level* modules.
The test files in the first example (:ref:`src layout <src-layout>`) would be imported as
``test_app`` and ``test_view`` top-level modules by adding ``tests/`` to ``sys.path``.
This results in a drawback compared to the import mode ``importlib``:
your test files must have **unique names**.
If you need to have test modules with the same name,
as a workaround you might add ``__init__.py`` files to your ``tests`` folder and subfolders,
changing them to packages:
..code-block::text
pyproject.toml
mypkg/
...
tests/
__init__.py
foo/
__init__.py
test_view.py
bar/
__init__.py
test_view.py
Now pytest will load the modules as ``tests.foo.test_view`` and ``tests.bar.test_view``,
allowing you to have modules with the same name.
But now this introduces a subtle problem:
in order to load the test modules from the ``tests`` directory,
pytest prepends the root of the repository to ``sys.path``,
which adds the side-effect that now ``mypkg`` is also importable.
This is problematic if you are using a tool like tox_ to test your package in a virtual environment,
because you want to test the *installed* version of your package,
not the local code from the repository.
The ``importlib`` import mode does not have any of the drawbacks above,
because ``sys.path`` is not changed when importing test modules.
.._`buildout`: http://www.buildout.org/en/latest/
.._`use tox`:
@@ -263,8 +273,8 @@ tox
Once you are done with your work and want to make sure that your actual
package passes all tests you may want to look into :doc:`tox <tox:index>`, the
virtualenv test automation tool and its :doc:`pytest support <tox:example/pytest>`.
tox helps you to setup virtualenv environments with pre-defined
virtualenv test automation tool.
``tox`` helps you to setup virtualenv environments with pre-defined
dependencies and then executing a pre-configured test command with
options. It will run tests against the installed package and not
against your source code checkout, helping to detect packaging
@@ -286,3 +296,20 @@ See also `pypa/setuptools#1684 <https://github.com/pypa/setuptools/issues/1684>`
setuptools intends to
`remove the test command <https://github.com/pypa/setuptools/issues/931>`_.
Checking with flake8-pytest-style
---------------------------------
In order to ensure that pytest is being used correctly in your project,
it can be helpful to use the `flake8-pytest-style <https://github.com/m-burst/flake8-pytest-style>`_ flake8 plugin.
flake8-pytest-style checks for common mistakes and coding style violations in pytest code,
such as incorrect use of fixtures, test function names, and markers.
By using this plugin, you can catch these errors early in the development process
and ensure that your pytest code is consistent and easy to maintain.
A list of the lints detected by flake8-pytest-style can be found on its `PyPI page <https://pypi.org/project/flake8-pytest-style/>`_.
..note::
flake8-pytest-style is not an official pytest project. Some of the rules enforce certain style choices, such as using `@pytest.fixture()` over `@pytest.fixture`, but you can configure the plugin to fit your preferred style.
pytest as a testing framework needs to import test modules and ``conftest.py`` files for execution.
Importing files in Python (at least until recently) is a non-trivial processes, often requiring
changing :data:`sys.path`. Some aspects of the
Importing files in Python is a non-trivial processes, so aspects of the
import process can be controlled through the ``--import-mode`` command-line flag, which can assume
these values:
*``prepend`` (default): the directory path containing each module will be inserted into the *beginning*
of :py:data:`sys.path` if not already there, and then imported with the :func:`__import__ <__import__>` builtin.
.._`import-mode-prepend`:
This requires test module names to be unique when the test directory tree is not arranged in
packages, because the modules will put in :py:data:`sys.modules` after importing.
*``prepend`` (default): the directory path containing each module will be inserted into the *beginning*
of :py:data:`sys.path` if not already there, and then imported with
the :func:`importlib.import_module <importlib.import_module>` function.
It is highly recommended to arrange your test modules as packages by adding ``__init__.py`` files to your directories
containing tests. This will make the tests part of a proper Python package, allowing pytest to resolve their full
name (for example ``tests.core.test_core`` for ``test_core.py`` inside the ``tests.core`` package).
If the test directory tree is not arranged as packages, then each test file needs to have a unique name
compared to the other test files, otherwise pytest will raise an error if it finds two tests with the same name.
This is the classic mechanism, dating back from the time Python 2 was still supported.
.._`import-mode-append`:
*``append``: the directory containing each module is appended to the end of :py:data:`sys.path` if not already
there, and imported with ``__import__``.
there, and imported with :func:`importlib.import_module <importlib.import_module>`.
This better allows to run test modules against installed versions of a package even if the
package under test has the same import root. For example:
@@ -38,23 +46,78 @@ these values:
the tests will run against the installed version
of ``pkg_under_test`` when ``--import-mode=append`` is used whereas
with ``prepend`` they would pick up the local version. This kind of confusion is why
we advocate for using :ref:`src <src-layout>` layouts.
we advocate for using :ref:`src-layouts <src-layout>`.
Same as ``prepend``, requires test module names to be unique when the test directory tree is
not arranged in packages, because the modules will put in :py:data:`sys.modules` after importing.
*``importlib``: new in pytest-6.0, this mode uses :mod:`importlib` to import test modules. This gives full control over the import process, and doesn't require changing :py:data:`sys.path`.
.._`import-mode-importlib`:
For this reason this doesn't require test module names to be unique, but also makes test
modules non-importable by each other.
*``importlib``: this mode uses more fine control mechanisms provided by :mod:`importlib` to import test modules, without changing :py:data:`sys.path`.
Advantages of this mode:
* pytest will not change :py:data:`sys.path` at all.
* Test module names do not need to be unique -- pytest will generate a unique name automatically based on the ``rootdir``.
Disadvantages:
* Test modules can't import each other.
* Testing utility modules in the tests directories (for example a ``tests.helpers`` module containing test-related functions/classes)
are not importable. The recommendation in this case it to place testing utility modules together with the application/library
code, for example ``app.testing.helpers``.
Important: by "test utility modules" we mean functions/classes which are imported by
other tests directly; this does not include fixtures, which should be placed in ``conftest.py`` files, along
with the test modules, and are discovered automatically by pytest.
It works like this:
1. Given a certain module path, for example ``tests/core/test_models.py``, derives a canonical name
like ``tests.core.test_models`` and tries to import it.
For non-test modules this will work if they are accessible via :py:data:`sys.path`, so
for example ``.env/lib/site-packages/app/core.py`` will be importable as ``app.core``.
This is happens when plugins import non-test modules (for example doctesting).
If this step succeeds, the module is returned.
For test modules, unless they are reachable from :py:data:`sys.path`, this step will fail.
2. If the previous step fails, we import the module directly using ``importlib`` facilities, which lets us import it without
changing :py:data:`sys.path`.
Because Python requires the module to also be available in :py:data:`sys.modules`, pytest derives a unique name for it based
on its relative location from the ``rootdir``, and adds the module to :py:data:`sys.modules`.
For example, ``tests/core/test_models.py`` will end up being imported as the module ``tests.core.test_models``.
..versionadded:: 6.0
..note::
Initially we intended to make ``importlib`` the default in future releases, however it is clear now that
it has its own set of drawbacks so the default will remain ``prepend`` for the foreseeable future.
..note::
By default, pytest will not attempt to resolve namespace packages automatically, but that can
be changed via the :confval:`consider_namespace_packages` configuration variable.
..seealso::
The :confval:`pythonpath` configuration variable.
The :confval:`consider_namespace_packages` configuration variable.
:ref:`test layout`.
We intend to make ``importlib`` the default in future releases, depending on feedback.
``prepend`` and ``append`` import modes scenarios
-------------------------------------------------
Here's a list of scenarios when using ``prepend`` or ``append`` import modes where pytest needs to
change ``sys.path`` in order to import test modules or ``conftest.py`` files, and the issues users
change :py:data:`sys.path` in order to import test modules or ``conftest.py`` files, and the issues users
might encounter because of that.
Test modules / ``conftest.py`` files inside packages
@@ -83,7 +146,7 @@ pytest will find ``foo/bar/tests/test_foo.py`` and realize it is part of a packa
there's an ``__init__.py`` file in the same folder. It will then search upwards until it can find the
last folder which still contains an ``__init__.py`` file in order to find the package *root* (in
this case ``foo/``). To load the module, it will insert ``root/`` to the front of
``sys.path`` (if not there already) in order to load
:py:data:`sys.path` (if not there already) in order to load
``test_foo.py`` as the *module*``foo.bar.tests.test_foo``.
The same logic applies to the ``conftest.py`` file: it will be imported as ``foo.conftest`` module.
@@ -113,8 +176,8 @@ When executing:
pytest will find ``foo/bar/tests/test_foo.py`` and realize it is NOT part of a package given that
there's no ``__init__.py`` file in the same folder. It will then add ``root/foo/bar/tests`` to
``sys.path`` in order to import ``test_foo.py`` as the *module*``test_foo``. The same is done
with the ``conftest.py`` file by adding ``root/foo`` to ``sys.path`` to import it as ``conftest``.
:py:data:`sys.path` in order to import ``test_foo.py`` as the *module*``test_foo``. The same is done
with the ``conftest.py`` file by adding ``root/foo`` to :py:data:`sys.path` to import it as ``conftest``.
For this reason this layout cannot have test modules with the same name, as they all will be
imported in the global import namespace.
@@ -127,7 +190,7 @@ Invoking ``pytest`` versus ``python -m pytest``
-----------------------------------------------
Running pytest with ``pytest [...]`` instead of ``python -m pytest [...]`` yields nearly
equivalent behaviour, except that the latter will add the current directory to ``sys.path``, which
equivalent behaviour, except that the latter will add the current directory to :py:data:`sys.path`, which
@@ -46,7 +44,7 @@ There are several limitations and difficulties with this approach:
2. parametrizing the "db" resource is not straight forward:
you need to apply a "parametrize" decorator or implement a
:py:func:`~hookspec.pytest_generate_tests` hook
:hook:`pytest_generate_tests` hook
calling :py:func:`~pytest.Metafunc.parametrize` which
performs parametrization at the places where the resource
is used. Moreover, you need to modify the factory to use an
@@ -94,15 +92,14 @@ Direct parametrization of funcarg resource factories
Previously, funcarg factories could not directly cause parametrization.
You needed to specify a ``@parametrize`` decorator on your test function
or implement a ``pytest_generate_tests`` hook to perform
or implement a :hook:`pytest_generate_tests` hook to perform
parametrization, i.e. calling a test multiple times with different value
sets. pytest-2.3 introduces a decorator for use on the factory itself:
..code-block::python
@pytest.fixture(params=["mysql","pg"])
defdb(request):
...# use request.param
defdb(request):...# use request.param
Here the factory will be invoked twice (with the respective "mysql"
and "pg" values set as ``request.param`` attributes) and all of
@@ -143,8 +140,7 @@ argument:
..code-block::python
@pytest.fixture()
defdb(request):
...
defdb(request):...
The name under which the funcarg resource can be requested is ``db``.
@@ -153,8 +149,7 @@ aka:
..code-block::python
defpytest_funcarg__db(request):
...
defpytest_funcarg__db(request):...
But it is then not possible to define scoping and parametrization.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
