(diff better viewed ignoring whitespace)
In pytest<8, the collection tree for `pyargs` arguments in an invocation
like this:
pytest --collect-only --pyargs pyflakes.test.test_undefined_names
looked like this:
```
<Package test>
<Module test_undefined_names.py>
<UnitTestCase Test>
<TestCaseFunction test_annotationUndefined>
... snipped ...
```
The pytest 8 collection improvements changed it to this:
```
<Dir pytest>
<Dir .tox>
<Dir venv>
<Dir lib>
<Dir python3.11>
<Dir site-packages>
<Package pyflakes>
<Package test>
<Module test_undefined_names.py>
<UnitTestCase Test>
<TestCaseFunction test_annotationUndefined>
... snipped ...
```
Besides being egregious (and potentially even worse than the above,
going all the way to the root, for system-installed packages, as is
apparently common in CI), this also caused permission errors when trying
to probe some of those intermediate directories.
This change makes `--pyargs` arguments no longer try to add parent
directories to the collection tree according to the `--confcutdir` like
they're regular arguments. Instead, only add the parents that are in the
import path. This now looks like this:
```
<Package .tox/venv/lib/python3.11/site-packages/pyflakes>
<Package test>
<Module test_undefined_names.py>
<UnitTestCase Test>
<TestCaseFunction test_annotationUndefined>
... snipped ...
```
Fix#11904.
Allow for the output of test case execution to be controlled independently from the application verbosity level.
`verbosity_test_case` is the new ini setting to adjust this functionality.
Fix#11639
Passing a short path in the command line was causing the matchparts check to fail, because ``Path(short_path) != Path(long_path)``.
Using ``os.path.samefile`` as fallback ensures the comparsion works on Windows when comparing short/long paths.
Fix#11895
Fix#12021.
Reopens#11706.
This reverts commit 12b9bd5801.
This change caused a bad regression in pytest-xdist:
https://github.com/pytest-dev/pytest-xdist/issues/1024
pytest-xdist necessarily has special handling of `--maxfail` and session
fixture teardown get executed multiple times with the change.
Since I'm not sure how to adapt pytest-xdist myself, revert for now.
I kept the sticky `shouldstop`/`shouldfail` changes as they are good
ideas regardless I think.
Dicts these days preserve order, so the sort is no longer needed to
achieve determinism.
As shown by the `test_dynamic_parametrized_ordering` test, this can
change the ordering of items, but only in equivalent ways (same number
of setups/teardowns per scope), it will just respect the user's given
ordering better (hence `vxlan` items now ordered before `vlan` items
compared to the previous ordering).
Test:
`warnings.warn()` expects that its first argument is a `str` or a
`Warning`, but since 9454fc38d3
`pytest.warns()` no longer allows `Warning` instances unless the first
argument the `Warning` was initialized with is a `str`. Furthermore, if
the `Warning` was created without arguments then `pytest.warns()` raises
an unexpected `IndexError`. The new tests reveal the problem.
Fix:
`pytest.warns()` now allows using `warnings.warn()` with a `Warning`
instance, as is required by Python, with one exception. If the warning
used is a `UserWarning` that was created by passing it arguments and the
first argument was not a `str` then `pytest.raises()` still considers
that an error. This is because if an invalid type was used in
`warnings.warn()` then Python creates a `UserWarning` anyways and it
becomes impossible for `pytest` to figure out if that was done
automatically or not.
[ran: rebased on previous commit]
Today `pyproject.toml` is the standard for declaring a Python project root, so seems reasonable to consider it for the ini configuration (and specially `rootdir`) in case we do not find other suitable candidates.
Related to #11311
Previously this would trigger an `AssertionError`.
While it could be considered a bug-fix, but given it now can be relied upon, it is probably better to consider it an improvement.
Fix#11311
* Improve error message when using @pytest.fixture twice
While obvious in hindsight, this error message confused me. I thought my fixture
function was used in a test function twice, since the wording is ambiguous.
Also, the error does not tell me *which* function is the culprit.
Finally, this adds a test, which wasn't done in
cfd16d0dac where this was originally implemented.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Fix#11929.
Figured out what's going on. We have the following collection tree:
```
<Dir pyspacewar>
<Dir src>
<Package pyspacewar>
<Package tests>
<DoctestModule test_main.py>
<DoctestItem pyspacewar.tests.test_main.doctest_main>
```
And the `test_main.py` contains an autouse fixture (`fake_game_ui`) that
`doctest_main` needs in order to run properly. The fixture doesn't run!
It doesn't run because nothing collects the fixtures from (calls
`parsefactories()` on) the `test_main.py` `DoctestModule`.
How come it only started happening with commit
ab63ebb3dc07b89670b96ae97044f48406c44fa0? Turns out it mostly only
worked accidentally. Each `DoctestModule` is also collected as a normal
`Module`, with the `Module` collected after the `DoctestModule`. For
example, if we add a non-doctest test to `test_main.py`, the collection
tree looks like this:
```
<Dir pyspacewar>
<Dir src>
<Package pyspacewar>
<Package tests>
<DoctestModule test_main.py>
<DoctestItem pyspacewar.tests.test_main.doctest_main>
<Module test_main.py>
<Function test_it>
```
Now, `Module` *does* collect fixtures. When autouse fixtures are
collected, they are added to the `_nodeid_autousenames` dict.
Before ab63ebb3dc, `DoctestItem` consults
`_nodeid_autousenames` at *setup* time. At this point, the `Module` has
collected and so it ended up picking the autouse fixture (this relies on
another "accident", that the `DoctestModule` and `Module` have the same
node ID).
After ab63ebb3dc, `DoctestItem` consults
`_nodeid_autousenames` at *collection* time (= when it's created). At
this point, the `Module` hasn't collected yet, so the autouse fixture is
not picked out.
The fix is simple -- have `DoctestModule.collect()` call
`parsefactories`. From some testing I've done it shouldn't have negative
consequences (I hope).
ruff is faster and handle everything we had prior.
isort configuration done based on the indication from
https://github.com/astral-sh/ruff/issues/4670, previousely based on
reorder-python-import (#11896)
flake8-docstrings was a wrapper around pydocstyle (now archived) that
explicitly asks to use ruff in https://github.com/PyCQA/pydocstyle/pull/658.
flake8-typing-import is useful mainly for project that support python 3.7
and the one useful check will be implemented in https://github.com/astral-sh/ruff/issues/2302
We need to keep blacken-doc because ruff does not handle detection
of python code inside .md and .rst. The direct link to the repo is
now used to avoid a redirection.
Manual fixes:
- Lines that became too long
- % formatting that was not done automatically
- type: ignore that were moved around
- noqa of hard to fix issues (UP031 generally)
- fmt: off and fmt: on that is not really identical
between black and ruff
- autofix re-order in pre-commit from faster to slower
Co-authored-by: Ran Benita <ran@unusedvar.com>
The current version (0.23.4) explicitly does not support pytest 8 yet, so we fallback to the previous release in the hope that at least our integration tests pass.
Change our mypy configuration to disallow untyped defs by default, which ensures *new* files added to the code base are fully typed.
To avoid having to type-annotate everything now, add `# mypy: allow-untyped-defs` to files which are not fully type annotated yet.
As we fully type annotate those modules, we can then just remove that directive from the top.
Refs #11662.
--- Problem
Each fixture definition has a "visibility", the `FixtureDef.baseid`
attribute. This is nodeid-like string. When a certain `node` requests a
certain fixture name, we match node's nodeid against the fixture
definitions with this name.
The matching currently happens on the *textual* representation of the
nodeid - we split `node.nodeid` to its "parent nodeids" and then check
if the fixture's `baseid` is in there.
While this has worked so far, we really should try to avoid textual
manipulation of nodeids as much as possible. It has also caused problem
in an odd case of a `Package` in the root directory: the `Package` gets
nodeid `.`, while a `Module` in it gets nodeid `test_module.py`. And
textually, `.` is not a parent of `test_module.py`.
--- Solution
Avoid this entirely by just checking the node hierarchy itself. This is
made possible by the fact that we now have proper `Directory` nodes
(`Dir` or `Package`) for the entire hierarchy.
Also do the same for `_getautousenames` which is a similar deal.
The `iterparentnodeids` function is no longer used and is removed.
* Fix handling empty values of NO_COLOR and FORCE_COLOR
Fix handling NO_COLOR and FORCE_COLOR environment variables to correctly
be ignored when they are set to an empty value, as defined
in the specification:
> Command-line software which adds ANSI color to its output by default
> should check for a NO_COLOR environment variable that, when present
> *and not an empty string* (regardless of its value), prevents
> the addition of ANSI color.
(emphasis mine, https://no-color.org/)
The same is true of FORCE_COLOR, https://force-color.org/.
* Streamline testing for FORCE_COLOR and NO_COLOR
Streamline the tests for FORCE_COLOR and NO_COLOR variables, and cover
all possible cases (unset, set to empty, set to "1"). Combine the two
assert functions into one taking boolean parameters. Mock file.isatty
in all circumstances to ensure that the environment variables take
precedence over the fallback value resulting from isatty check (or that
the fallback is actually used, in the case of both FORCE_COLOR
and NO_COLOR being unset).
* Put a 'reset' color in front of the highlighting
When doing the highlighting, some lexers will not set the initial color
explicitly, which may lead to the red from the errors being propagated
to the start of the expression
* Add syntactic highlighting to the error explanations
This updates the various error reporting to highlight python code when
displayed, to increase readability and make it easier to understand
Previously the error report would have all sections glued together:
- The assertion representation
- The error explanation
- The full diff
This makes it hard to see at a glance where which one starts and ends.
One of the representation (dataclasses, tuples, attrs) does display a
newlines at the start already.
Let's add a newlines before the error explanation and before the full
diff, so we get an easier to read report.
This has one disadvantage: we get one line less in the least verbose
mode, where the output gets truncated.
The default for `_prepareconfig` is to use `sys.argv`, which in this
case are the flags passed to (top-level) `pytest`. This is not the
intention, the tests themselves should not be affected by it.
When running `tox -e py-lsof` I get a deluge of this warning:
```
src/pytest/.tox/py-lsof-numpy-pexpect/lib/python3.11/site-packages/_pytest/pytester.py:130: EncodingWarning: UTF-8 Mode affects locale.getpreferredencoding(). Consider locale.getencoding() instead.
```
Use `locale.getencoding` instead.
The normal default pretty printer is not great when objects are nested
and it can get hard to read the diff.
Instead, provide a pretty printer that behaves more like when json get
indented, which allows for smaller, more meaningful differences, at
the expense of a slightly longer diff.
This does not touch the other places where the pretty printer is used,
and only updated the full diff one.
Reset color-related environment variables in a fixture to prevent them
from affecting test results. Otherwise, some of the tests fail
e.g. if NO_COLOR is set in the calling environment.
The tested py.path.local.make_numbered_dir function is *not*
multiprocess safe, because is uses os.listdir which itself is not.
The os.listdir documentation explicitly states that:
> If a file is removed from or added to the directory during the call
> of this function, whether a name for that file be included is unspecified.
This can lead to a race when:
1. process A attempts to create directory N
2. the creation fails, as another process already created it in the meantime
3. process A calls listdir to determine a more recent maxnum
4. processes B+ repeatedly create newer directories and they delete directory N
5. process A doesn't have directory N or any newer directory in listdir result
6. process A attempts to create directory N again and raises
For details, see https://github.com/pytest-dev/pytest/issues/11603#issuecomment-1805708144
and bellow.
Additionally, the test itself has a race in batch_make_numbered_dirs.
When this functions attempts to write to repro-N/foo,
repro-N may have already been removed by another process.
For details, see https://github.com/pytest-dev/pytest/issues/11603#issuecomment-1804714313
and bellow.
---
The tested py.path.local.make_numbered_dir function is not used in pytest.
There is a different implementation in _pytest.pathlib.
Closes#11603
Logging has many global states, and we did foresee this by creating a ``cleanup_disabled_logging`` fixture,
however one might still forget to use it and failures leak later -- sometimes not even in the same PR, because the order
of the tests might change in the future, specially when running under xdist.
This problem surfaced during pytest-dev/pytest#11530, where tests unrelated to the change started to fail.
The current method as the following problem, described by Sadra
Barikbin:
The tests that request both `pytester` and `monkeypatch` and use
`monkeypatch.chdir` without context, relying on `monkeypatch`'s teardown
to restore cwd. This doesn't work because the following sequence of
actions take place:
- `monkeypatch` is set up.
- `pytester` is set up. It saves the original cwd and changes it to a
new one dedicated to the test function.
- Test function calls `monkeypatch.chdir()` without context.
`monkeypatch` saves cwd, which is not the original one, before
changing it.
- `pytester` is torn down. It restores the cwd to the original one.
- `monkeypatch` is torn down. It restores cwd to what it has saved.
The solution here is to have pytester use `monkeypatch.chdir()` itself,
then everything is handled correctly.
Tests if a captured exception group contains an expected exception.
Will raise `AssertionError` if the wrapped exception is not an exception group.
Supports recursive search into nested exception groups.
Starting with `resolve_package_path` and its associated tests,
this refactoring seeks to make path concatenation more
readable and consistent within tests/functions.
As discussed in #11413:
- code is free to use either `/` and `joinpath`
- consistency within a function is more important than consistency across the codebase
- it is nice to use `/` when it is more readable
- it is nice to use `joinpath` when there is little context
- be mindful that `joinpath` may be clearer when joining multiple segments
For packages, `import_path` receives the path to the package's `__init__.py` file, however module names (as they live in `sys.modules`) should not include the `__init__` part.
For example, `app/core/__init__.py` should be imported as `app.core`, not as `app.core.__init__`.
Fix#11306
Move handling of user_properties to `finalize()`.
Previously if a fixture failed during teardown, `pytest_runtest_logreport` would not be called with "teardown", resulting in the user properties not being saved on the JUnit XML file.
Fixes: #11367
Bruno Oliveira
Ran Benita
dependabot[bot]
Patrick Lannigan
robotherapist
Bruno Oliveira
Eric Larson
Ben Leith
Reagan Lee
Eero Vaher
Pierre Sassoulas
Florian Bruhin
whysage
Clément Robert
Joachim B Haga
clee2000
Franck Charras
Fabian Sturm
Ben Brown
Michał Górny
Benjamin Schubert
Jens Tröger
Miro Hrončok
Sharad Nair
Benjamin Schubert
ryanpudd
Tanya Agarwal
Chris Mahoney
Mihail Milushev
Mihail Milushev
Aleksandr Brodin
Warren Markham
Marc Mueller
Sadra Barikbin
Ronny Pfannschmidt
Israel Fruchter
Sean Malloy
Jon Parise
Zac Hatfield-Dodds