This indicates at least for people using type checkers that these
classes are not designed for inheritance and we make no stability
guarantees regarding inheritance of them.
Currently this doesn't show up in the docs. Sphinx does actually support
`@final`, however it only works when imported directly from `typing`,
while we import from `_pytest.compat`.
In the future there might also be a `@sealed` decorator which would
cover some more cases.
Usually, we use semi-colon punctuation mark to connect closely related
ideas. Sentences which are after semicolon, begins with small letter,
and last sentence always ends with a period mark, see "Garner's Modern
American Usage" book for more information about usage of punctuation
mark. So removing punctuation mark altogether is a good idea, as
@gnikonorov suggested [1].
[1]: https://github.com/pytest-dev/pytest/pull/7760#pullrequestreview-489232607
For decorated functions, the lineno of the FunctionDef AST node points
to the `def` line, not to the first decorator line. On the other hand,
in code objects, the `co_firstlineno` points to the first decorator
line.
Assertion rewriting inserts some imports to code it rewrites. The
imports are inserted at the lineno of the first statement in the AST. In
turn, the code object compiled from the rewritten AST uses the lineno of
the first statement (which is the first inserted import).
This means that given a module like this,
```py
@foo
@bar
def baz(): pass
```
the lineno of the code object without assertion rewriting
(`--assertion=plain`) is 1, but with assertion rewriting it is 3.
And *this* causes some issues for the exception repr when e.g. the
decorator line is invalid and raises during collection. The code becomes
confused and crashes with
INTERNALERROR> File "_pytest/_code/code.py", line 638, in get_source
INTERNALERROR> lines.append(space_prefix + source.lines[line_index].strip())
INTERNALERROR> IndexError: list index out of range
Fix it by special casing decorators. Maybe there are other cases like
this but off hand I can't think of another Python construct where the
lineno of the item would be after its first line, and this is the only
such issue we have had reported.
- be more vague about "packaging metadata" over explicitly mentioning `setup.py`
(such that `pyproject.toml`-based distributions are allowed)
- drop extensions on `README.txt` / `LICENSE.txt` (it's more common to have `.md`
/ `.rst` / no extension)
- remove duplicate mention of license packaging metadata
The release-on-comment script is always executed on *master*, so we should
execute the `release.py` script using tox to ensure we create the
right environment.
Also fixed errors in the error handling code.
When a name is exported from `pytest`, prefer to refer to it by that
rather than its `_pytest` import path. It is shorter and more
appropriate in user-facing documentation (although that's not really
visible).
Our plan is to expose more names for typing purposes, in which can this
could be more comprehensive.
Before, `PluginManager` was a copy of the pluggy doc, and
`PytestPluginManager` was documented separately.
pytest users only really need to know about `PytestPluginManager`, so
instead of splitting have the `PytestPluginManager` documentation
include all of `PluginManager` members and remove `PluginManager` from
the reference (it is still shown as the base class).
Warnings are a central part of Python, so much that Python itself has
command-line and environtment variables to handle warnings.
By moving the concept of warning handling into Config, it becomes natural to
filter warnings issued as early as possible, even before the "_pytest.warnings"
plugin is given a chance to spring into action. This also avoids the weird
coupling between config and the warnings plugin that was required before.
Fix#6681Fix#2891Fix#7620Fix#7626Close#7649
Co-authored-by: Ran Benita <ran@unusedvar.com>
Inline `_makeitem()` so that `self.ihook` (which is moderately
expensive) can be called only once.
Note: the removed test "test_makeitem_non_underscore" comes from an old
behavior of skipping names that start with `_` which has since been
generalized, making the test no longer relevant.
When a Python object (module/class/instance) is collected, for each name
in `obj.__dict__` (and up its MRO) the pytest_pycollect_makeitem hook is
called for potentially creating a node for it.
These Python objects have a bunch of builtin attributes that are
extremely unlikely to be collected. But due to their pervasiveness,
dispatching the hook for them ends up being mildly expensive and also
pollutes PYTEST_DEBUG=1 output and such.
Let's just ignore these attributes.
On the pandas test suite commit 04e9e0afd476b1b8bed930e47bf60e,
collect only, irrelevant lines snipped, about 5% improvement:
Before:
```
51195095 function calls (48844352 primitive calls) in 39.089 seconds
ncalls tottime percall cumtime percall filename:lineno(function)
226602/54 0.145 0.000 38.940 0.721 manager.py:90(_hookexec)
72227 0.285 0.000 20.146 0.000 python.py:424(_makeitem)
72227 0.171 0.000 16.678 0.000 python.py:218(pytest_pycollect_makeitem)
```
After:
```
48410921 function calls (46240870 primitive calls) in 36.950 seconds
ncalls tottime percall cumtime percall filename:lineno(function)
181429/54 0.113 0.000 36.777 0.681 manager.py:90(_hookexec)
27054 0.130 0.000 17.755 0.001 python.py:465(_makeitem)
27054 0.121 0.000 16.219 0.001 python.py:218(pytest_pycollect_makeitem)
```
In ff8b7884e8 NOTSET was changed to a
singleton enum, which ended up unexpectedly triggering a code path in ID
generation which checks for `isinstance(Enum)`.
Add an explicit case for it, which is not too bad anyway.
* Update talks/trainings page
- Remove past webinar
- Add new open training
- Add some talks/webinars by Oliver Bestwalter and by me
- Remove some stale link targets
* Move sidebar to index
This is a more sensible interface for matchnodes.
This also fixes a sort-of bug where a recursive call to matchnodes
raises NoMatch which would terminate the entire tree, even if other
branches may find a match. Though I don't think it's actually possible.
This reverts commit f10ab021e2.
The commit was good in that it removed a non-trivial amount of code
duplication. However it was done in the wrong layer (nodes.py) and split
up a major part of the collection (the filesystem traversal) to a
separate class making it harder to understand.
We should try to reduce the duplication, but in a more appropriate
manner.
The path part of a `<path>::part1::part2` style collection argument must
be a file, not a directory.
Previously this crashed with an uncool assert "invalid arg".
The `CaptureManager.global_and_fixture_disabled()` context manager (and
`CaptureFixture.disabled()` which calls it) did `suspend(); ...;
resume()` but if the capturing was already suspended, the `resume()`
would resume it when it shouldn't.
This caused caused some messages to be swallowed when `--log-cli` is
used because it uses `global_and_fixture_disabled` when capturing is not
necessarily resumed.
This fixes an issue where pylint complains about missing implementations
of abstract methods in subclasses of `File` which only override
`collect()` (as they should).
It is also cleaner and makes sense, these methods really don't need to
be overridden.
The previous methods defined directly on `FSCollector` and `Package` are
deprecated, to be removed in pytest 7.
See commits e2934c3f8c and
f10ab021e2 for reference.
and include a demo of toml 'literal strings'
Update doc/en/warnings.rst
Apply suggestion by Ran
Fix linting
Co-authored-by: Bruno Oliveira <nicoddemus@gmail.com>
Didn't call it absolute or absolute_path to avoid conflicts with
possible variable names.
Didn't call it abspath to avoid confusion with os.path.abspath.
This makes mypy raise an error whenever it detects code which is
statically unreachable, e.g.
x: int
if isinstance(x, str):
... # Statement is unreachable [unreachable]
This is really neat and finds quite a few logic and typing bugs.
Sometimes the code is intentionally unreachable in terms of types, e.g.
raising TypeError when a function is given an argument with a wrong
type. In these cases a `type: ignore[unreachable]` is needed, but I
think it's a nice code hint.
This is a follow up to 3f8200676f which didn't
make it clear that log_print is also removed in the changelog and didn't remove
it from the reference docs.
There are some ones we *would* like to enforce, like
D401 First line should be in imperative mood
but have too many false positives, so I left them out.
This prevents referring to a generic type without filling in its generic
type parameters.
The FixtureDef typing might need some more refining in the future.
In Python, if module A defines a name `name`, and module B does `import
name from A`, then another module C can `import name from B`.
Sometimes it is intentional -- module B is meant to "reexport" `name`.
But sometimes it is just confusion/inconsistency on where `name` should
be imported from.
mypy has a flag `--no-implicit-reexport` which puts some order into
this. A name can only be imported from a module if
1. The module defines the name
2. The module's `__all__` includes the name
3. The module imports the name as `from ... import .. as name`.
This flag is included in mypy's `--strict` flag.
I like this flag, but I realize it is a bit controversial, and in
particular item 3 above is a bit unfriendly to contributors who don't
know about it. So I didn't intend to add it to pytest.
But while investigating issue 7589 I came upon mypy issue 8754 which
causes `--no-implicit-reexport` to leak into installed libraries and
causes some unexpected typing differences *in pytest* if the user uses
this flag.
Since the diff mostly makes sense, let's just conform to it.
We barely use it; the couple places that do are not really worth the
extra dependency, I think the code is clearer without it.
Also simplifies one (regular) itertools usage.
Also improves a check and an error message in `pytest.raises`.
Part of the effort to reduce dependency on the py library.
Besides that, py.xml implements its own XML serialization which is
pretty scary.
I tried to keep the code with minimal changes (though it could use some
cleanups). The differences in behavior I have noticed are:
- Attributes in the output are not sorted.
- Some unneeded escaping is no longer performed, for example escaping
`"` to `"` in a text node.
1. Remove sys.maxunicode check & comment. Nowadays it is always a
constant 0x10ffff.
2. Pre-generate the pattern. Possible due to 1.
3. Compile the regex lazily. No reason to pay startup cost for it.
4. Add docstring in particular to explain a subtle point.
`os.scandir()`, introduced in Python 3.5, is much faster than
`os.listdir()`. See https://www.python.org/dev/peps/pep-0471/.
It also has a `DirEntry` which can be used to further reduce syscalls in
some cases.
Part of reducing dependency on `py`. Also enables upcoming improvements.
In cases where there are simpler alternatives (in tests), I used those.
What's left are a couple of uses in `_pytest.main` and `_pytest.python`
and they only have modest requirements, so all of the featureful code
from py is not needed.
This fixes CI on Windows since GitHub Actions started installing WSL on
their images which apparently installs some wrapper `bash` which does
not run actual bash.
Previously was restricted to >=1.8.2,<2.1, but newer versions have some
nice improvements I'd like to be able to use in upcoming changes.
Changelog: https://www.sphinx-doc.org/en/master/changes.html#release-3-1-0-released-jun-08-2020
There are two issues that came up:
1. `highlightlang` is deprecated for `highlight`.
2. Doesn't like having two `automethod` generated for the same
`Metafunc.parametrize` method. Gives this warning:
`pytest/doc/en/reference.rst:846: WARNING: duplicate object description of _pytest.python.Metafunc.parametrize, other instance in reference, use :noindex: for one of them`
To work around this I make `pytest.mark.parametrize` link to
`Metafunc.parametrize` instead of repeating it.
The previous typing had an object passed to the user, which they can't
do anything with without asserting, which is inconvenient. Change it to
Any instead.
Note that what comes *back* to pytest (the return value) should be an
`object`, because we want to handle arbitrary objects without assuming
anything about them.
If a test runtest phase (not setup) dynamically adds a pytest.mark.xfail
mark to the item, it should be respected, but it wasn't. This regressed
in 3e6fe92b7e (not released).
Fix it by just always refreshing the mark if needed. This is mostly what
was done before but in a more roundabout way.
This introduces a new role, `:globalvar:`, so we can mark/reference
variables like `pytest_plugins`, `pytestmark`, etc. This besides being useful
also makes the documentation look more consistent.
eval() is used for evaluating string conditions in skipif/xfail e.g.
@pytest.mark.skipif("1 == 0")
This is the only code that uses `_pytest._code.compile()`, so removing
its last use enables us to remove it entirely.
In this case it doesn't add much. Plain compile() gives a good enough
error message.
For regular exceptions, the message is the same.
For SyntaxError exceptions, e.g. "1 ==", the previous code adds a little
bit of useful context:
```
invalid syntax (skipping.py:108>, line 1)
The above exception was the direct cause of the following exception:
1 ==
^
(code was compiled probably from here: <0-codegen /pytest/src/_pytest/skipping.py:108>) (line 1)
During handling of the above exception, another exception occurred:
Error evaluating 'skipif' condition
1 ==
^
SyntaxError: invalid syntax
```
The new code loses it:
```
unexpected EOF while parsing (<skipif condition>, line 1)
During handling of the above exception, another exception occurred:
Error evaluating 'skipif' condition
1 ==
^
SyntaxError: invalid syntax
```
Since the old message is a minor improvement to an unlikely error
condition in a deprecated feature, I think it is not worth all the code
that it requires.
This has been there since as far as the git history goes (2007), is not
covered by any test, and says "Buggy python version consider upgrading".
Hopefully everyone have upgraded...
Setting log_level via the CLI or .ini will control the log level of the
report that is dumped upon failure of a test.
If caplog modified the log level during the execution of that test, it
should not impact the level that is displayed upon failure in the
"captured log report" section.
[
ran:
- rebased
- reused handler
- changed store keys also to "caplog_handler_*"
- added changelog
all bugs are mine :)
]
There is no need to do the XPASS check here, pytest_runtest_makereport
already handled that (the current handling there is dead code).
All the hook needs to do is refresh the xfail evaluation if needed, and
check the NOTRUN condition again.
Previously, skipif/xfail marks were evaluated using a `MarkEvaluator`
class. I found this class very difficult to understand.
Instead of `MarkEvaluator`, rewrite using straight functions which are
hopefully easier to follow.
I tried to keep the semantics exactly as before, except improving a few
error messages.
This type was actually in `_pytest.skipping` previously, but was moved to
`_pytest.mark.evaluate` in cf40c0743c.
I think the previous location was more appropriate, because the
`MarkEvaluator` is not a generic mark facility, it is explicitly and
exclusively used by the `skipif` and `xfail` marks to evaluate their
particular set of arguments. So it is better to put it in the plugin
code.
Putting `skipping` related functionality into the core `_pytest.mark`
module also causes some import cycles which we can avoid.
`@pytest.mark.xfail` is meant to work with arbitrary items, and there is
a test `test_mark_xfail_item` which verifies this.
However, the code for some reason uses `pytest_pyfunc_call` for the
call phase check, which only works for Function items. The test
mentioned above only passed "accidentally" because the
`pytest_runtest_makereport` hook also runs a `evalxfail.istrue()` which
triggers and evaluation, but conceptually it shouldn't do that.
Change to `pytest_runtest_call` to make the xfail checking properly
generic.
While working on improving the documentation of the
`pytest_runtest_setup` hook, I came up with this text:
> Called to perform the setup phase of the test item.
>
> The default implementation runs ``setup()`` on item and all of its
> parents (which haven't been setup yet). This includes obtaining the
> values of fixtures required by the item (which haven't been obtained
> yet).
But upon closer inspection I noticed this line at the start of
`SetupState.prepare` (which is what does the actual work for
`pytest_runtest_setup`):
self._teardown_towards(needed_collectors)
which implies that the setup phase of one item might trigger teardowns
of *previous* items. This complicates the simple explanation. It also
seems like a completely undesirable thing to do, because it breaks
isolation between tests -- e.g. a failed teardown of one item shouldn't
cause the failure of some other items just because it happens to run
after it.
So the first thing I tried was to remove that line and see if anything
breaks -- nothing did. At least pytest's own test suite runs fine. So
maybe it's just dead code?
This option checks even functions which are not annotated. It's a good
step to ensure that existing type annotation are correct.
In a Pareto fashion, the last few holdouts are always the ugliest,
beware.
The FixtureFunctionMarker attrs class already converts the params
itself.
When adding types, the previous converter composition causes some type
error, but extracting it to a standalone function fixes the issue (a
lambda is not supported by the mypy plugin, currently).
* Fix typo in capture.rst documentation
Rename ``capfsysbinary`` to ``capsysbinary`` as the former does not exist as far as i can see.
* Make Python uppercase in doc/en/capture.rst
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
* Remove the sentence entirely
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Co-authored-by: Ran Benita <ran@unusedvar.com>
* enable tox -e docs natively on windows using a wrapper
* rename the towncrier script; run the towncrier command in a safer manner
* use subprocess.call; call exit() around main on towncrier wrapper
* change to sys.exit() instead of builtin exit()
There are state transitions start/done/suspend/resume and two additional
operations snap/writeorg.
Previously it was not well defined in what order they can be called, and
which operations are idempotent.
Formalize this and enforce using assert checks with informative error
messages if they fail (rather than random AttributeErrors).
Make it easier to read the file in progression, and avoid forward
references for upcoming type annotations.
There is one cycle, CaptureManager <-> CaptureFixture, which is hard to
untangle.
(This commit should be added to `.gitblameignore`).
`TerminalWriter`, imported recently from `py`, contains its own
incomplete wcwidth (`char_with`/`get_line_width`) implementation. The
`TerminalReporter` also needs this, but uses the external `wcwidth`
package.
This commit brings the `TerminalWriter` implementation up-to-par with
`wcwidth`, moves to implementation to a new file `_pytest._io.wcwidth`
which is used everywhere, and removes the dependency.
The differences compared to the `wcwidth` package are:
- Normalizes the string before counting.
- Uses Python's `unicodedata` instead of vendored Unicode tables. This
means the data corresponds to the Python's version Unicode version
instead of the `wcwidth`'s package version.
- Apply some optimizations.
pytest.fixture() can be used either as
@pytest.fixture
def func(): ...
or as
@pytest.fixture()
def func(): ...
or (while maybe not intended)
func = pytest.fixture(func)
so it needs to inspect internally whether it got a function in the first
positional argument or not.
Previously, there were was oddity. In the following,
func = pytest.fixture(func, autouse=True)
# OR
func = pytest.fixture(func, parms=['a', 'b'])
The result is as if `func` wasn't passed.
There isn't any reason for this special that I can understand, so remove
it.
_pytest.timing is an indirection to 'time' functions, which pytest production
code should use instead of 'time' directly.
'mock_timing' is a new fixture which then mocks those functions, allowing us
to write time-reliable tests which run instantly and are not flaky.
This was triggered by recent flaky junitxml tests on Windows related to timing
issues.
This is already done in pytest_runtest_logstart, so the fspath is
already guaranteed to have been printed (for xdist, it is disabled
anyway).
write_fspath_result is mildly expensive so it is worth avoiding calling
it twice.
The `FDCapture`/`FDCaptureBinary` classes, used by `capfd`/`capfdbinary`
fixtures and the `--capture=fd` option (set by default), redirect FDs
1/2 (stdout/stderr) to a temporary file. To do this, they need to save
the old file by duplicating the FD before redirecting it, to be restored
once finished.
Previously, if this duplicating (`os.dup()`) failed, most likely due to
that FD being invalid, the FD redirection would silently not be done. The
FD capturing also performs python-level redirection (monkeypatching
`sys.stdout`/`sys.stderr`) which would still be done, but direct writes
to the FDs would fail.
This is not great. If pytest is run with `--capture=fd`, or a test is
using `capfd`, it expects writes to the FD to work and be captured,
regardless of external circumstances.
So, instead of disabling FD capturing, keep the redirection to a
temporary file, just don't restore it after closing, because there is
nothing to restore to.
This makes a difference for e.g. pytest-xdist:
Before:
```
--dist=distmode set mode for distributing tests to exec environments. each: …
available environment. loadscope: …
grouped by file to any available environment. (default) no: …
```
After:
```
--dist=distmode set mode for distributing tests to exec environments.
each: send each test to all available environments.
load: load balance by sending any pending test to any available environment.
…
(default) no: run tests inprocess, don't distribute.
```
This might also result in unexpected changes (hard wrapping), when line
endings where used unintentionally, e.g. with:
```
help="""
some long
help text
"""
```
But the benefits from that are worth it, and it is easy to fix, as will
be done for the internal `assertmode` option.
Currently, a bad logging call, e.g.
logger.info('oops', 'first', 2)
triggers the default logging handling, which is printing an error to
stderr but otherwise continuing.
For regular programs this behavior makes sense, a bad log message
shouldn't take down the program. But during tests, it is better not to
skip over such mistakes, but propagate them to the user.
Previously, a LoggingCaptureHandler was instantiated for each test's
setup/call/teardown which turns out to be expensive.
Instead, only keep one instance and reset it between runs.
The tests came via c629f6b18 and c61ff31ffa.
The fixes from there are kind of obsoleted by 4cd08f9 (moving to importlib),
but it makes sense to keep them as integration tests in general.
The logstart/logreport/logfinish hooks don't need the stuff in
_runtest_for. The test capturing catching_logs call is irrelevant for
them, and the item-conditional sections are gone.
- Instead of making it optional, always set up a handler, but possibly
going to /dev/null. This simplifies the code by removing a lot of
conditionals. It also can replace the NullHandler() we already add.
- Change `set_log_path` to just change the stream, instead of recreating
one. Besides plugging a resource leak, it enables the next item.
- Remove the capturing_logs from _runtest_for, since it sufficiently
covered by the one in pytest_runtestloop now, which wraps all other
_runtest_for calls.
The first item alone would have had an adverse performance impact, but
the last item removes it.
Remove usage of `@contextmanager` as it is a bit slower than
hand-rolling, and also disallows re-entry which we want to use.
Removing protections around addHandler()/removeHandler(), because
logging already checks that internally.
Conceptually it doesn't check per catching_logs (and catching_logs
doesn't restore the older one either). It is just something that is
defined for each handler once.
The default message is often hard to read:
E _pytest.config.ConftestImportFailure: (local('D:\\projects\\pytest\\.tmp\\root\\foo\\conftest.py'), (<class 'RuntimeError'>, RuntimeError('some error',), <traceback object at 0x000001CCC3E39348>))
Using a shorter message is better:
E _pytest.config.ConftestImportFailure: RuntimeError: some error (from D:\projects\pytest\.tmp\root\foo\conftest.py)
And we don't really lose any information due to exception chaining.
This removes the KeyError from the traceback chain when an
conftest fails to import:
return self._conftestpath2mod[key]
E KeyError: WindowsPath('D:/projects/pytest/.tmp/root/foo/conftest.py')
During handling of the above exception, another exception occurred:
...
raise RuntimeError("some error")
E RuntimeError: some error
During handling of the above exception, another exception occurred:
...
E _pytest.config.ConftestImportFailure: (...)
By slightly changing the code, we can remove the first chain, which is often
very confusing to users and doesn't help with anything.
Fix#7223
Only filter with known failures, and explicitly keep paths of passed
arguments.
This also displays the "run-last-failure" status before collected files,
and does not update the cache with "--collect-only".
Fixes https://github.com/pytest-dev/pytest/issues/6968.
The previous commit made this possible, so utilize it.
Since legacy.py becomes pretty bare, I inlined it into __init__.py. I'm
not sure it's really "legacy" anyway!
Using a simple 50000 items benchmark with `--collect-only -k nomatch`:
Before (two commits ago):
======================== 50000 deselected in 10.31s =====================
19129345 function calls (18275596 primitive calls) in 10.634 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.001 0.001 2.270 2.270 __init__.py:149(pytest_collection_modifyitems)
1 0.036 0.036 2.270 2.270 __init__.py:104(deselect_by_keyword)
50000 0.055 0.000 2.226 0.000 legacy.py:87(matchkeyword)
After:
======================== 50000 deselected in 9.37s =========================
18029363 function calls (17175972 primitive calls) in 9.701 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 1.394 1.394 __init__.py:239(pytest_collection_modifyitems)
1 0.057 0.057 1.393 1.393 __init__.py:162(deselect_by_keyword)
The matching itself can be optimized more but that's a different story.
In current pytest, the same expression is matched against all items. But
it is re-parsed for every match.
Add support for "compiling" an expression and reusing the result. Errors
may only occur during compilation.
This is done by parsing the expression into a Python `ast.Expression`,
then `compile()`ing it into a code object. Evaluation is then done using
`eval()`.
Note: historically we used to use `eval` directly on the user input --
this is not the case here, the expression is entirely under our control
according to our grammar, we just JIT-compile it to Python as a
(completely safe) optimization.
New errors:
testing/test_setupplan.py:104:15: E741 ambiguous variable name 'l'
testing/test_setupplan.py:107:15: E741 ambiguous variable name 'l'
extra/get_issues.py:48:29: E741 ambiguous variable name 'l'
testing/test_error_diffs.py:270:32: E741 ambiguous variable name 'l'
Not so sure about it but easier to just fix.
But more importantly, is a large amount of typing-related issues there
were fixed which necessitated noqa's which can now be removed.
The `-k '-expr'` syntax is an old alias to `-k 'not expr'`. It's also
not a very convenient to have syntax that start with `-` on the CLI.
Deprecate it and suggest replacing with `not`.
---
The `-k 'expr:'` syntax discards all items until the first match and
keeps all subsequent, e.g. `-k foo` with
test_bar
test_foo
test_baz
results in `test_foo`, `test_baz`. That's a bit weird, so deprecate it
without a replacement. If someone complains we can reconsider or devise
a better alternative.
Running `pytest | head -1` and similar causes an annoying error to be
printed to stderr:
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe
(or possibly even a propagating exception in older/other Python versions).
The standard UNIX behavior is to handle the EPIPE silently. To
recommended method to do this in Python is described here:
https://docs.python.org/3/library/signal.html#note-on-sigpipe
It is not appropriate to apply this recommendation to `pytest.main()`,
which is used programmatically for in-process runs. Hence, change
pytest's entrypoint to a new `pytest.console_main()` function, to be
used exclusively by pytest's CLI, and add the SIGPIPE code there.
Fixes#4375.
When setting up the warnings capture, filter strings (with the general
form `action:message:category:module:line`) are collected from the
cmdline, ini and item and applied. This happens for every test and other
cases.
To apply a string it needs to be parsed into a tuple, and it turns out
this is slow. Since we already vendor the parsing code from Python's
warnings.py, we can speed it up by caching the result. After splitting
the parsing part from the applying part, the parsing is pure and is
straightforward to cache.
An alternative is to parse ahead of time and reuse the result, however
the caching solution turns out cleaner and more general in this case.
On this benchmark:
import pytest
@pytest.mark.parametrize("x", range(5000))
def test_foo(x): pass
Before:
============================ 5000 passed in 14.11s =============================
14365646 function calls (13450775 primitive calls) in 14.536 seconds
After:
============================ 5000 passed in 13.61s =============================
13290372 function calls (12375498 primitive calls) in 14.034 seconds
- replace "tests with warnings" with just warnings: they a) might not
come from a test in the first place, and b) a single test might have
multiple warnings.
- fix usage of (unused) argument to `collapsed_location_report`
Co-authored-by: Ran Benita <ran@unusedvar.com>
Move {Passthrough,CaptureIO} to capture module, and rename Passthrough
-> Tee to match the existing terminology.
Co-authored-by: Ran Benita <ran@unusedvar.com>
Also delay calling tearDown() when --pdb is given, so users still have
access to the instance variables (which are usually cleaned up during tearDown())
when debugging.
Fix#6947
Instead of trying to handle unittest-async functions in pytest_pyfunc_call,
let the unittest framework handle them instead.
This lets us remove the hack in pytest_pyfunc_call, with the upside that
we should support any unittest-async based framework.
Also included 'asynctest' as test dependency for py37-twisted, and renamed
'twisted' to 'unittestextras' to better reflect that we install 'twisted' and
'asynctest' now.
This also fixes the problem of cleanUp functions not being properly called
for async functions.
Fix#7110Fix#6924
Previously, the expressions given to the `-m` and `-k` options were
evaluated with `eval`. This causes a few issues:
- Python keywords cannot be used.
- Constants like numbers, None, True, False are not handled correctly.
- Various syntax like numeric operators and `X if Y else Z` is supported
unintentionally.
- `eval()` is somewhat dangerous for arbitrary input.
- Can fail in many ways so requires `except Exception`.
The format we want to support is quite simple, so change to a custom
parser. This fixes the issues above, and gives us full control of the
format, so can be documented comprehensively and even be extended in the
future if we wish.
Currently this property is computed eagerly, which means
get_line_width() is computed on everything written, but that is a slow
function.
Compute it lazily, so that get_line_width() only runs when needed.
Flushing on every write is somewhat expensive.
Rely on line buffering instead (if line buffering for stdout is
disabled, there must be some reason...), and add explicit flushes when
not outputting lines.
This is how regular `print()` e.g. work so should be familiar.
The code used an O(n^2) loop. Replace list with set to make it O(n).
For backward compatibility the filesystem cache still remains a list.
On this test:
import pytest
@pytest.mark.parametrize("x", range(5000))
def test_foo(x): pass
run with `pytest --collect-only`:
Before: 0m1.251s
After: 0m0.921s
Remove this message which was shown in the warning summary of pytest's
own testsuite:
testing/test_cacheprovider.py::TestNewAPI::test_cache_writefail_cachfile_silent
testing/test_cacheprovider.py:40: PytestCacheWarning: could not create cache path /tmp/pytest-of-ran/pytest-2/test_cache_writefail_cachfile_silent0/.pytest_cache/v/test/broken
cache.set("test/broken", [])
Currently this test issues a warning which is displayed in the warning
summary (of pytest's own test suite):
testing/acceptance_test.py::TestGeneralUsage::test_early_skip
/tmp/pytest-of-ran/pytest-396/test_early_skip0/conftest.py:2: PytestDeprecationWarning: The pytest_collect_directory hook is not working.
Please use collect_ignore in conftests or pytest_collection_modifyitems.
def pytest_collect_directory():
I think the filter was meant to be `ignore` in the first place, and not
`always` which is not a valid action AFAIK.
This function is exposed and kept alive for the oejskit plugin which is
abandoned and no longer works with recent plugins, so let's prepare to
completely remove it.
Earlier pytest-bot would only print out the exception in cases of
failure but did not provide context on failing command and error from
command. This patch adds the errors from the command to the exception
message.
`Command` provides abstraction over the command to run and helps
in collecting errors from the first failing command only.
With this, we don't need to check `returncode` from each command that
we run, we can run all the commands and will have access to the error
from the first command that failed. This pattern was taken from Go.
Please refer: https://blog.golang.org/errors-are-values
The `_pytest._code._reprcompare` that was referred to previously doesn't
exist -- it was moved to other places but wasn't updated. This regressed
in f423ce9c01. Now we don't want it
anymore, so keep the status quo by explicitly removing them.
* tests: fix TypeError with test_mark_closest
It fails when trying to run it actually:
> TypeError: test_has_inherited() takes 0 positional arguments but 1 was given
* Fix testing/test_collection.py::TestCollector::test_getparent
Mostly I wanted to get rid of mentions of "MarkItem" which is something
that no longer exists, but I improved a little beyond that and annotated
some simple types.
It was surprising that `tryfirst=True` would not result in lines being
added to the beginning with `pytest_report_header`.
This is due to lines being reversed, and therefore the same applies to
`pytest_report_collectionfinish`.
TestDurations tests the `--durations=N` functionality which reports N
slowest tests, with durations <= 0.005s not shown by default.
The test relies on real time.sleep() (in addition to the code which uses
time.perf_counter()) which makes it flaky and inconsistent between
platforms.
Instead of trying to tweak it more, make it use fake time instead. The
way it is done is a little hacky but seems to work.
This hook has some functionality to provide explicit markup for the test
status. It seemed unused and wasn't tested, so I was tempted to remove
it, but I found that the pytest-rerunfailures plugin uses it, so
document it and add a test instead.
Use `testdir.syspathinsert()` with multiprocessing tests:
- test_chained_exceptions_no_reprcrash
- test_exception_handling_no_traceback
This only works currently because `_importtestmodule` changes `sys.path`
as a side-effect.
It appears to be only required on Windows though - likely due to the
multiprocessing method used there.
The dynamic scope feature added in 10bf6aac76
necessitated some wrangling of arguments in pytest.fixture(). In
particular, it deprecated positional arguments in favor of keyword-only
arguments, while keeping backward compatibility.
The way it did this avoided some code duplication but ended up being
quite hard to follow and to annotate with types.
Replace it with some straightforward code, which is not very DRY but is
simple and easy to remove when the time comes.
This changes the link anchors in "reference.html", from e.g.
`reference.html#pytest-current-test` to
`reference.html#envvar-PYTEST_CURRENT_TEST`, but I think that is OK, and
not worth adding labels for the old anchors.
twisted started to use `attr.s(eq)` argument which was added recently,
so it fails with oldattrs. One of the CI jobs ran twisted and oldattrs
together, so it started to fail.
Move the twisted code to be covered by another job, and remove it from
the job with the oldattrs.
Previously, writing to sys.stdout/stderr in text-mode (e.g.
`print('foo')`) while a `capsysbinary` fixture is active, would crash
with:
/usr/lib/python3.7/contextlib.py:119: in __exit__
next(self.gen)
E TypeError: write() argument must be str, not bytes
This is due to some confusion in the types. The relevant functions are
`snap()` and `writeorg()`. The function `snap()` returns what was
captured, and the return type should be `bytes` for the binary captures
and `str` for the regular ones. The `snap()` return value is eventually
passed to `writeorg()` to be written to the original file, so it's input
type should correspond to `snap()`. But this was incorrect for
`SysCaptureBinary`, which handled it like `str`.
To fix this, be explicit in the `snap()` and `writeorg()`
implementations, also of the other Capture types.
We can't add type annotations yet, because the current inheritance
scheme breaks Liskov Substitution and mypy would complain. To be
refactored later.
Fixes: https://github.com/pytest-dev/pytest/issues/6871
Co-authored-by: Ran Benita (some modifications & commit message)
I tried to understand what the `safe_text_dupfile()` function and
`EncodedFile` class do. Outside tests, `EncodedFile` is only used by
`safe_text_dupfile`, and `safe_text_dupfile` is only used by
`FDCaptureBinary.__init__()`. I then started to eliminate always-true
conditions based on the single call site, and in the end nothing was
left except of a couple workarounds that are still needed.
The commit which added the checks for os.dup a15afb5e48
suggests it was done for Jython. But pytest doesn't support Jython
anymore (Jython is Python 2 only).
Furthermore, it looks like the faulthandler plugin (bundled in pytest
and enabled by default) uses os.dup() unprotected and there have not
been any complaints.
So seems better to just remove these checks, and only add if someone
with a legitimate use case complains.
`KeywordMapping` returns a bool on lookup which when passed to eval
fail on certain operations such as index access and attribute access.
We catch all exceptions and raise a `UsageError`.
- it should not document the deprecated `result`; used the same as
pluggy documents itself
- add a "hookwrapper" label, that could be used by pluggy (currently it
links to the section)
- use pluggy's `hookwrappers` label for linking to its documentation
When using `(i)var` in the class docstring it would link `duration` to
`_pytest.runner.TestReport.duration`.
This moves the docstrings to the attributes properly.
This started from fixing the test, where `"xdist_strict True"` was used
as a single argument, although you typically would see `["xdist_strict",
"True"]`.
Improves the error message to mention the option that caused the error.
pytest has several instances where plugins set their own attributes on
objects they receive in hooks, like nodes and config. Since plugins are
detached from these object's definition by design, this causes a problem
for type checking because these attributes are not defined and mypy
complains.
Fix this by giving these objects a "store" which can be used by plugins
in a type-safe manner.
Currently this mechanism is private. We can consider exposing it at a
later point.
The pytest-konira plugin has not seen an update since 2011, moreover
the "project description" on PyPI points to a dubious website that
does not actually correspond to the project and instead redirects to
advertising content.
This was changed unintentionally in 45c4a8fb3 (pytest 5.3.0), but only
EnvironmentErrors might have `errno`.
Since that is not really guaranteed and it is good to have more
information this uses the string representation of the exc in the trace
message.
It should only return `True` when something is to be ignored, not
`False` otherwise typically.
This caused e.g. bad interaction with the cacheprovider (before
https://github.com/pytest-dev/pytest/pull/6448).
Passing in a tuple crashes in `_prepareconfig`:
def test_invoke_with_tuple(self):
> pytest.main(("-h",))
src/_pytest/config/__init__.py:82: in main
config = _prepareconfig(args, plugins)
src/_pytest/config/__init__.py:229: in _prepareconfig
return pluginmanager.hook.pytest_cmdline_parse(
…
src/_pytest/helpconfig.py:98: in pytest_cmdline_parse
config = outcome.get_result() # type: Config
src/_pytest/config/__init__.py:808: in pytest_cmdline_parse
self.parse(args)
src/_pytest/config/__init__.py:1017: in parse
self._preparse(args, addopts=addopts)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
def _preparse(self, args: List[str], addopts: bool = True) -> None:
…
if addopts:
ini_addopts = self.getini("addopts")
if ini_addopts:
> args[:] = self._validate_args(ini_addopts, "via addopts config") + args
E TypeError: can only concatenate list (not "tuple") to list
addopts = True
args = ('-h',)
env_addopts = ''
ini_addopts = ['-rfEX', …]
src/_pytest/config/__init__.py:956: TypeError: can only concatenate list (not "tuple") to list
Might be worth handling (converting it to a list for example), but it
was documented to be a list to begin with when removing support for
strings (a7e401656).
Remove `_tmp_path_factory` and `_tmpdirhandler` from the config object.
- `_tmpdirhandler` has been deprecated since 2.8.0 (0f52856f9), when
`tmpdir_factory` has been added.
- `_tmp_path_factory` should have probably never been added there in the
first place, but maybe just used the same pattern (16e2737da).
This better reflects the inheritance / smartness with regard to raw or
encoded.
- FDCaptureBinary
- FDCapture
- SysCaptureBinary
- SysCapture
- TeeSysCapture
Mypy currently is unable to handle assigning attributes on function:
https://github.com/python/mypy/issues/2087.
pytest uses this for the outcome exceptions -- `pytest.fail.Exception`,
`pytest.exit.Exception` etc, and this is the canonical name by which they
are referred.
Initially we started working around this with type: ignores, and later
by switching e.g. `pytest.fail.Exception` with the direct exception
`Failed`. But this causes a lot of churn and is not as nice. And I also
found that some code relies on it, in skipping.py:
def pytest_configure(config):
if config.option.runxfail:
# yay a hack
import pytest
old = pytest.xfail
config._cleanup.append(lambda: setattr(pytest, "xfail", old))
def nop(*args, **kwargs):
pass
nop.Exception = xfail.Exception
setattr(pytest, "xfail", nop)
...
So it seems better to support it. Use a hack to make it work. The rest
of the commit rolls back all of the workarounds we added up to now.
`pytest.raises.Exception` also exists, but it's not used much so I kept
it as-is for now.
Hopefully in the future mypy supports this and this ugliness can be
removed.
Allows for filtering of PytestCacheWarning.
Using `_issue_warning_captured` is not necessary here, and was probably
only used because the cacheprovider misses warnings during
`pytest_sessionfinish`, which is also fixed here.
I think the usage of `_issue_warning_captured` can be removed/reduced
further, but also that this is good enough for now.
Ref: https://github.com/pytest-dev/pytest/issues/6681.
* Turn ReprTraceback into attrs class
* Use attr.asdict with serialize_repr_{crash,traceback}
* Turn ReprFileLocation into attrs class, convert py.path.local
* Use code highlighting if pygments is installed
* Use colorama constants instead of bare ascii codes
Could not find the exact equivalent of 'hl-reset' code using colorama
constants though.
* Refactor ASCII color handling into a fixture
* Revert back to using explicit color codes
* In Python 3.5 skip rest of tests that require ordered markup in colored output
Use suggestion in review and use a subplugin so hooks will only be active
if we enable faulthandler ourselves.
Fix#6575
Co-authored-by: Daniel Hahler <git@thequod.de>
ExitCode is used in several internal modules and hooks and so with type
annotations added, needs to be imported a lot.
_pytest.main, being the entry point, generally sits at the top of the
import tree.
So, it's not great to have ExitCode defined in _pytest.main, because it
will cause a lot of import cycles once type annotations are added (in
fact there is already one, which this change removes).
Move it to _pytest.config instead.
_pytest.main still imports ExitCode, so importing from there still
works, although external users should really be importing from `pytest`.
The convention is "assert result is expected". Pytest's error diffs now
reflect this. "-" means that sth. expected is missing in the result and
"+" means that there are unexpected extras in the result.
Fixes: #3333
Fix mypy errors:
src/_pytest/runner.py:36: error: "addoption" of "OptionGroup" does not return a value [func-returns-value]
src/_pytest/helpconfig.py:64: error: "addoption" of "OptionGroup" does not return a value [func-returns-value]
src/_pytest/terminal.py:67: error: "_addoption" of "OptionGroup" does not return a value [func-returns-value]
src/_pytest/terminal.py:75: error: "_addoption" of "OptionGroup" does not return a value [func-returns-value]
Fixes collection error with Python 3.5.3 (Travis):
testing/test_parseopt.py:2: in <module>
import distutils.spawn
.tox/py35-coverage/lib/python3.5/distutils/__init__.py:4: in <module>
import imp
.tox/py35-coverage/lib/python3.5/imp.py:33: in <module>
PendingDeprecationWarning, stacklevel=2)
E PendingDeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
Build log: https://travis-ci.org/blueyed/pytest/builds/648305304
This started by looking at how to get the current test item in general,
and then I noticed that it is not necessary for the capture plugin to
track it manually in the first place.
Previously this would be turned via `py.path.local("")` into the current
working directory.
This appears to be what `fspath = fn and py.path.local(fn) or None`
tries to avoid in `getfslineno`'s `TypeError` handling already, if
`Code` would raise it.
Everything was using `_pytest.compat.getfslineno` basically, which
wrapped `_pytest._code.source.getfslineno`.
This moves the extra code from there into it directly, and uses the
latter everywhere.
This helps to eventually remove the one in compat eventually, and also
causes less cyclic imports.
* Explicitly state on the PR template that we can squash commits
This way we don't need to ask every time, and users who for some reason
would not like us to squash their commits can explicitly state so.
Plugins specified with ``-p`` are now loaded after internal plugins, which
results in their hooks being called *before* the internal ones.
This makes the ``-p`` behavior consistent with ``PYTEST_PLUGINS``.
* fix/adjust test_disable_plugin_autoload
* adjust test_plugin_loading_order
`Generator` was removed in 7eb28f9eb, and this pleases mypy to correctly
complain that `FunctionMixin` has no `config` (within
`_prunetraceback`).
* typing: _prunetraceback
* minor: imports, typing
Reverts https://github.com/pytest-dev/pytest/pull/6610.
The tested `getfslineno` is `src/_pytest/_code/source.py` actually,
exported via `src/_pytest/_code/__init__.py`.
I've confused it with the one in `src/_pytest/compat.py` apparently.
It should be in `test_code` when testing `_pytest._code.getfslineno`,
not to be confused with `_pytest._code.source.getfslineno`.
Adds an extra assert (via https://github.com/pytest-dev/pytest/pull/6590).
This often might be causing for jobs to take longer than 10 minutes,
which is a timeout Codecov uses to wait for successful CI.
Also it is good in general to have CI finish faster, of course.
From the description it seemed to me as if just prefixing your methods with ``test_`` was enough, but you also need to prefix your class with ``Test``. Of course, in the reference material this is clearly stated, but I think it makes sense to mention it here as well, since you also mention the part about the methods' prefix.
While it might not help with the following, it certainly might happen as
well.
```
+ curl -S -L --connect-timeout 5 --retry 6 -s https://codecov.io/bash -o codecov-upload.sh
curl: (16) Error in the HTTP2 framing layer
```
- refactor _get_main_color/build_summary_stats_line
- factor out property _is_last_item; test_summary_stats: tr._is_last_item
- _write_progress_information_filling_space: remove color arg
- use setter for stats, handling main color
- _get_main_color: skip cache for last item
- Handle random order in test for py35.
Mypy complains about this once the function is typed:
src/_pytest/main.py:85: error: "_addoption" of "OptionGroup" does not return a value
src/_pytest/main.py:133: error: "addoption" of "OptionGroup" does not return a value
Previously, this cache was used with 3 different and mutually exclusive
key-type -> value-type combinations. Mypy can't properly type this. It's
also quite confusing.
Split to 3 different dicts instead.
Previously, _initialparts was a list whose first item was a
`py.path.local` and the rest were `str`s. This is not something that
mypy is capable of modeling. The type `List[Union[str, py.path.local]]`
is too broad and would require asserts for every access.
Instead, make each item a `Tuple[py.path.local, List[str]]`. This way
the structure is clear and the types are accurate.
To make sure any users who might have been accessing this (private)
field will not break silently, change the name to _initial_parts.
Make EncodedFile, used for captured output streams, method .write return
the number of characters written. Add test for captured stderr write.
Fixes#6557.
Co-Authored-By: Bruno Oliveira <nicoddemus@gmail.com>
Without restoring the cwd, successive tests might fail to parse the
config (via `_pytest.config._prepareconfig()`, for when `--lsof` is
used).
And it is good practice to restore the cwd in any case anyway.
These are more "dirty" than the previous batch (that's why they were
left out). The trouble is that `compile` can return either a code object
or an AST depending on a flag, so we need to add an overload to make the
common case Union free. But it's still worthwhile.
GitHub actions already should deploy.
This is a stop gap while we figure out why coverage dropped
when removing Travis builds in #6470
(cherry picked from commit d1d7e5d41b)
Somehow in Python 3.5 on Windows this test fails with:
File "c:\hostedtoolcache\windows\python\3.5.4\x64\Lib\multiprocessing\connection.py", line 302, in _recv_bytes
overlapped=True)
OSError: [WinError 6] The handle is invalid
This only happens in this platform and Python version, decided to use
a dummy traceback as originally done in #6412.
(cherry picked from commit b9c136b809)
This allows for e.g. Jedi to infer types (it checks the name).
It was only used to support Python 3.5.0/3.5.1, where this is is not
available in the `typing` module.
Ref: https://github.com/davidhalter/jedi/issues/1472
Uses `TYPE_CHECKING = False` in `_pytest.outcomes` to avoid having to
work around circular import.
Somehow in Python 3.5 on Windows this test fails with:
File "c:\hostedtoolcache\windows\python\3.5.4\x64\Lib\multiprocessing\connection.py", line 302, in _recv_bytes
overlapped=True)
OSError: [WinError 6] The handle is invalid
This only happens in this platform and Python version, decided to use
a dummy traceback as originally done in #6412.
This is different from what pre-commit (in "linting") runs in that it
uses stubs from (test) dependencies.
It would make sense to run this on CI additionally (since there is no
"pre-commit --skip mypy", and a separate config is not worth it).
But currently it triggers a false positive though anyway
(https://github.com/erikrose/more-itertools/pull/374).
This overwrites the `codecov.yml` file in the root of the repository with
`codecov-upstream.yml` file (which contains the code-cov token)´, so PRs
and branches on the repository can upload coverage.
Suggestion from here:
https://github.com/pytest-dev/pytest/pull/6421#issuecomment-571934112
Security concerns: the token might be misused, but only to upload bogus coverage
to `codecov.io`, so the team believe this is harmless. If we decide to fallback
from this decision , we just need to revoke the token.
Related to #6369
Both references were referencing links from Python because of our intersphinx
mapping in `conf.py`:
intersphinx_mapping = {"python": ("https://docs.python.org/3", None)}
Because Python's docs explicitly define both references, Sphinx fallbacks to
them instead of generating implicit references as was expected.
Fix#6397
Tracebacks coming from remote processes crated by the multiprocess module
will contain "RemoteTracebacks" which don't have a 'reprcrash' attribute
Fix#5971
I've updated the text and incorporated the topics from #5275, so this
can now be part of the official docs, and #5275 can be closed/unpinned.
Closes#5275
This fixes some type: ignores due to typeshed update.
Newer mypy seem to ignore unannotated functions better, so add a few
minor annotations so that existing correct type:ignores make sense.
* Rearrange section about context manager to be in order
* Link to `pytest.warns` and `recwarn` since a reader going top to bottom won't have seen about those yet.
* Used only context manager form in the example; the call form is somewhat obsolete
and is mentioned in the reference docs already.
* Reuse the 'myfunction' from first example on the second one
Co-Authored-By: Hugo van Kemenade <hugovk@users.noreply.github.com>
Co-Authored-By: Hugo van Kemenade <hugovk@users.noreply.github.com>
This includes our current full matrix (windows, linux and macos), for evaluting
purposes.
We should disconsider failures when evaluating PRs.
TODO:
- deploy
- coverage
- github release notes
Even with the above missing, I still believe it would be nice to merge
this and have GitHub actions working in parallel so we can evaluate performance
and usability from now on.
The order of invocations 'python -m pytest' and 'pytest' are different in the header and the explanation. Me being lazy reading about the behaviour of 'former' looked up quickly the title and rushed to implementation to discover it actually works the other way - as stated in the documentation. So I propose to switch the order in the title to achieve consistent ordering and not confusing somebody like me again! :)
While using pytest-repeat, I noticed the previous implementation is slow
for a large number of duplicate test ids. To optimize, this commit
reduces the amount of data copied and avoids using `in` with List
(unhashable type, and therefore is very slow for many elements).
As far as the output is concerned, they are both identical so it doesn't
make sense to have both.
setup, teardown, and collect failures are already reported as "errors", "E".
Now `tox -e docs` will also include the draft changelog for the
next version (locally only).
`CHANGELOG.rst` now only points to the changelog on READTHEDOCS so
sphinx diretives can be used.
Followup to https://github.com/pytest-dev/pytest/pull/6272
Sets `PY_COLORS=0` in the environment by default, which is used by pylib.
Via https://github.com/blueyed/pytest/pull/58
(initially cherry picked from commit f153ad33d10)
This feature was added in commit
007a77c2ba, but was never used in pytest
itself. A GitHub code search doesn't find any users either (only pytest
repo copies). It seems safe to clean up.
@@ -3,10 +3,13 @@ Thanks for submitting a PR, your contribution is really appreciated!
Here is a quick checklist that should be present in PRs.
- [ ] Target the `master` branch for bug fixes, documentation updates and trivial changes.
- [ ] Target the `features` branch for new features, improvements, and removals/deprecations.
- [ ] Include documentation when adding new features.
- [ ] Include new tests or update existing tests when applicable.
- [X] Allow maintainers to push and squash when merging my commits. Please uncheck this if you prefer to squash the commits yourself.
If this change fixes an issue, please:
- [ ] Add text like ``closes #XYZW`` to the PR description and/or commits (where ``XYZW`` is the issue number). See the [github docs](https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword) for more information.
Unless your change is trivial or a small documentation fix (e.g., a typo or reword of a small section) please:
or ``trivial``. You may skip creating the changelog entry if the change doesn't affect the
documented behaviour of pytest.
#. Add yourself to ``AUTHORS`` file if not there yet, in alphabetical order.
@@ -293,14 +319,13 @@ Here is a simple overview, with pytest-specific bits:
compare: your-branch-name
base-fork: pytest-dev/pytest
base: master # if it's a bugfix
base: features # if it's a feature
base: master
Writing Tests
----------------------------
~~~~~~~~~~~~~
Writing tests for plugins or for pytest itself is often done using the `testdir fixture <https://docs.pytest.org/en/latest/reference.html#testdir>`_, as a "black-box" test.
Writing tests for plugins or for pytest itself is often done using the `testdir fixture <https://docs.pytest.org/en/stable/reference.html#testdir>`_, as a "black-box" test.
For example, to ensure a simple test passes you can write:
@@ -337,16 +362,121 @@ one file which looks like a good fit. For example, a regression test about a bug
should go into ``test_cacheprovider.py``, given that this option is implemented in ``cacheprovider.py``.
If in doubt, go ahead and open a PR with your best guess and we can discuss this over the code.
Joining the Development Team
----------------------------
Anyone who has successfully seen through a pull request which did not
require any extra work from the development team to merge will
themselves gain commit access if they so wish (if we forget to ask please send a friendly
reminder). This does not mean your workflow to contribute changes,
reminder). This does not mean there is any change in your contribution workflow:
everyone goes through the same pull-request-and-review process and
no-one merges their own pull requests unless already approved. It does however mean you can
participate in the development process more fully since you can merge
pull requests from other contributors yourself after having reviewed
them.
Backporting bug fixes for the next patch release
------------------------------------------------
Pytest makes feature release every few weeks or months. In between, patch releases
are made to the previous feature release, containing bug fixes only. The bug fixes
usually fix regressions, but may be any change that should reach users before the
next feature release.
Suppose for example that the latest release was 1.2.3, and you want to include
a bug fix in 1.2.4 (check https://github.com/pytest-dev/pytest/releases for the
actual latest release). The procedure for this is:
#. First, make sure the bug is fixed the ``master`` branch, with a regular pull
request, as described above. An exception to this is if the bug fix is not
applicable to ``master`` anymore.
#.``git checkout origin/1.2.x -b backport-XXXX`` # use the master PR number here
#. Locate the merge commit on the PR, in the *merged* message, for example:
nicoddemus merged commit 0f8b462 into pytest-dev:master
#.``git cherry-pick -x -m1 REVISION`` # use the revision you found above (``0f8b462``).
#. Open a PR targeting ``1.2.x``:
* Prefix the message with ``[1.2.x]``.
* Delete the PR body, it usually contains a duplicate commit message.
Who does the backporting
~~~~~~~~~~~~~~~~~~~~~~~~
As mentioned above, bugs should first be fixed on ``master`` (except in rare occasions
that a bug only happens in a previous release). So who should do the backport procedure described
above?
1. If the bug was fixed by a core developer, it is the main responsibility of that core developer
to do the backport.
2. However, often the merge is done by another maintainer, in which case it is nice of them to
do the backport procedure if they have the time.
3. For bugs submitted by non-maintainers, it is expected that a core developer will to do
the backport, normally the one that merged the PR on ``master``.
4. If a non-maintainers notices a bug which is fixed on ``master`` but has not been backported
(due to maintainers forgetting to apply the *needs backport* label, or just plain missing it),
they are also welcome to open a PR with the backport. The procedure is simple and really
helps with the maintenance of the project.
All the above are not rules, but merely some guidelines/suggestions on what we should expect
about backports.
Handling stale issues/PRs
-------------------------
Stale issues/PRs are those where pytest contributors have asked for questions/changes
and the authors didn't get around to answer/implement them yet after a somewhat long time, or
the discussion simply died because people seemed to lose interest.
There are many reasons why people don't answer questions or implement requested changes:
they might get busy, lose interest, or just forget about it,
but the fact is that this is very common in open source software.
The pytest team really appreciates every issue and pull request, but being a high-volume project
with many issues and pull requests being submitted daily, we try to reduce the number of stale
issues and PRs by regularly closing them. When an issue/pull request is closed in this manner,
it is by no means a dismissal of the topic being tackled by the issue/pull request, but it
is just a way for us to clear up the queue and make the maintainers' work more manageable. Submitters
can always reopen the issue/pull request in their own time later if it makes sense.
When to close
~~~~~~~~~~~~~
Here are a few general rules the maintainers use to decide when to close issues/PRs because
of lack of inactivity:
* Issues labeled ``question`` or ``needs information``: closed after 14 days inactive.
* Issues labeled ``proposal``: closed after six months inactive.
* Pull requests: after one month, consider pinging the author, update linked issue, or consider closing. For pull requests which are nearly finished, the team should consider finishing it up and merging it.
The above are **not hard rules**, but merely **guidelines**, and can be (and often are!) reviewed on a case-by-case basis.
Closing pull requests
~~~~~~~~~~~~~~~~~~~~~
When closing a Pull Request, it needs to be acknowledge the time, effort, and interest demonstrated by the person which submitted it. As mentioned previously, it is not the intent of the team to dismiss stalled pull request entirely but to merely to clear up our queue, so a message like the one below is warranted when closing a pull request that went stale:
Hi <contributor>,
First of all we would like to thank you for your time and effort on working on this, the pytest team deeply appreciates it.
We noticed it has been awhile since you have updated this PR, however. pytest is a high activity project, with many issues/PRs being opened daily, so it is hard for us maintainers to track which PRs are ready for merging, for review, or need more attention.
So for those reasons we think it is best to close the PR for now, but with the only intention to cleanup our queue, it is by no means a rejection of your changes. We still encourage you to re-open this PR (it is just a click of a button away) when you are ready to get back to it.
Again we appreciate your time for working on this, and hope you might get back to this at a later time!
<bye>
Closing Issues
--------------
When a pull request is submitted to fix an issue, add text like ``closes #XYZW`` to the PR description and/or commits (where ``XYZW`` is the issue number). See the `GitHub docs <https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword>`_ for more information.
When an issue is due to user error (e.g. misunderstanding of a functionality), please politely explain to the user why the issue raised is really a non-issue and ask them to close the issue if they have no further questions. If the original requestor is unresponsive, the issue will be handled as described in the section `Handling stale issues/PRs`_ above.
The ``pytest`` framework makes it easy to write small tests, yet
scales to support complex functional testing for applications and libraries.
@@ -67,33 +71,33 @@ To execute it::
========================== 1 failed in 0.04 seconds ===========================
Due to ``pytest``'s detailed assertion introspection, only plain ``assert`` statements are used. See `getting-started <https://docs.pytest.org/en/latest/getting-started.html#our-first-test-run>`_ for more examples.
Due to ``pytest``'s detailed assertion introspection, only plain ``assert`` statements are used. See `getting-started <https://docs.pytest.org/en/stable/getting-started.html#our-first-test-run>`_ for more examples.
Features
--------
- Detailed info on failing `assert statements <https://docs.pytest.org/en/latest/assert.html>`_ (no need to remember ``self.assert*`` names);
- Detailed info on failing `assert statements <https://docs.pytest.org/en/stable/assert.html>`_ (no need to remember ``self.assert*`` names)
@@ -15,10 +15,10 @@ Each file should be named like ``<ISSUE>.<TYPE>.rst``, where
*``feature``: new user facing features, like new command-line options and new behavior.
*``improvement``: improvement of existing functionality, usually without requiring user intervention (for example, new fields being written in ``--junitxml``, improved colors in terminal, etc).
*``bugfix``: fixes a reported bug.
*``bugfix``: fixes a bug.
*``doc``: documentation improvement, like rewording an entire session or adding missing docs.
*``deprecation``: feature deprecation.
*``removal``: feature removal.
*``breaking``: a change which may break existing suites, such as feature removal or behavior change.
*``vendor``: changes in packages vendored in pytest.
*``trivial``: fixing a small typo or internal change that might be noteworthy.
@@ -31,6 +31,7 @@ changelog using that instead.
If you are not sure what issue type to use, don't hesitate to ask in your PR.
``towncrier`` preserves multiple paragraphs and formatting (code blocks, lists, and so on), but for entries
other than ``features`` it is usually better to stick to a single paragraph to keep it concise. You can install
``towncrier`` and then run ``towncrier --draft``
if you want to get a preview of how your change will look in the final release notes.
other than ``features`` it is usually better to stick to a single paragraph to keep it concise.
You can also run ``tox -e docs`` to build the documentation
with the draft changelog (``doc/en/_build/html/changelog.html``) if you want to get a preview of how your change will look in the final release notes.
There can be many different definitions of "success". Pytest can run many `nose and unittest`_ tests by default, so using pytest as your testrunner may be possible from day 1. Job done, right?
There can be many different definitions of "success". Pytest can run many nose_ and unittest_ tests by default, so using pytest as your testrunner may be possible from day 1. Job done, right?
It may be after the month is up, the partner project decides that pytest is not right for it. That's okay - hopefully the pytest team will also learn something about its weaknesses or deficiencies.
.._`nose and unittest`: faq.html#how-does-pytest-relate-to-nose-and-unittest
@@ -7,7 +7,7 @@ This is a bug-fix release, being a drop-in replacement. To upgrade::
pip install --upgrade pytest
The full changelog is available at http://doc.pytest.org/en/latest/changelog.html.
The full changelog is available at http://doc.pytest.org/en/stable/changelog.html.
Thanks to all who contributed to this release, among them:
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
