* adding --sw-skip shorthand for stepwise skip
* be explicit rather than implicit with default args for stepwise
* add constant for sw cache dir; only register plugin if necessary rather check check activity always;
* use str format; remove unused args in hooks
* assert cache upfront, allow stepwise to have a reference to the cache
* type hinting lf, skip, move literal strings into module constants
* convert parametrized option into a list
* add a sessionfinish hook for stepwise to keep backwards behaviour the same
* add changelog for #7938
* Improve performance of stepwise modifyitems & address PR feedback
* add test for stepwise deselected based on performance enhancements
* Apply suggestions from code review
* delete from items, account for edge case where failed_index = 0
Co-authored-by: Bruno Oliveira <nicoddemus@gmail.com>
In pytester tests, pytest stashes & restores the sys.modules for each
test. So if the test imports a new module, it is initialized anew each
time.
Turns out the readline module isn't multi-init safe, which causes
pytester.spawn to crash or hang. So preserve it as a workaround.
It turns out all autouse fixtures are kept in a global list, and thinned
out for a particular node using a linear scan of the entire list each
time.
Change the list to a dict, and only take the nodes we need.
ischildnode can be quite hot in some cases involving many fixtures.
However it is always used in a way that the nodeid is constant and the
baseid is iterated. So we can save work by pre-computing the parents of
the nodeid and use a simple containment test.
The `_getautousenames` function has the same stuff open-coded, so change
it to use the new function as well.
Since commit 0f918b1a9d pytest uses auto-generated autouse
pytest fixtures for the xunit fixtures
{setup,teardown}_{module,class,method,function}. All of these fixtures
were given the same name.
Unfortunately, pytest fixture lookup for a name works by grabbing all of
the fixtures globally declared with a name and filtering to only those
which match the specific node. So each xunit-using item iterates over a
list (of fixturedefs) of a size of all previous same-xunit-using items,
i.e. quadratic.
Fixing this properly to use a better data structure is likely to take
some effort, but we can avoid the immediate problem by just using
a different name for each item's autouse fixture, so it only matches
itself.
A benchmark is added to demonstrate the issue. It is still way too slow
after the fix and possibly still quadratic, but for a different reason
which is another matter.
Running --collect-only, before (snipped):
202533232 function calls (201902604 primitive calls) in 86.379 seconds
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 85.688 85.688 main.py:320(pytest_collection)
1 0.000 0.000 85.688 85.688 main.py:567(perform_collect)
80557/556 0.021 0.000 85.050 0.153 {method 'extend' of 'list' objects}
85001/15001 0.166 0.000 85.045 0.006 main.py:785(genitems)
10002 0.050 0.000 84.219 0.008 runner.py:455(collect_one_node)
10002 0.049 0.000 83.763 0.008 runner.py:340(pytest_make_collect_report)
10002 0.079 0.000 83.668 0.008 runner.py:298(from_call)
10002 0.019 0.000 83.541 0.008 runner.py:341(<lambda>)
5001 0.184 0.000 81.922 0.016 python.py:412(collect)
5000 0.020 0.000 81.072 0.016 python.py:842(collect)
30003 0.118 0.000 78.478 0.003 python.py:218(pytest_pycollect_makeitem)
30000 0.190 0.000 77.957 0.003 python.py:450(_genfunctions)
40001 0.081 0.000 76.664 0.002 nodes.py:183(from_parent)
30000 0.087 0.000 76.629 0.003 python.py:1595(from_parent)
40002 0.092 0.000 76.583 0.002 nodes.py:102(_create)
30000 0.305 0.000 76.404 0.003 python.py:1533(__init__)
15000 0.132 0.000 74.765 0.005 fixtures.py:1439(getfixtureinfo)
15000 0.165 0.000 73.664 0.005 fixtures.py:1492(getfixtureclosure)
15000 0.044 0.000 57.584 0.004 fixtures.py:1653(getfixturedefs)
30000 18.840 0.001 57.540 0.002 fixtures.py:1668(_matchfactories)
37507500 31.352 0.000 38.700 0.000 nodes.py:76(ischildnode)
15000 10.464 0.001 15.806 0.001 fixtures.py:1479(_getautousenames)
112930587/112910019 7.333 0.000 7.339 0.000 {built-in method builtins.len}
After:
51890333 function calls (51259706 primitive calls) in 27.306 seconds
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 26.783 26.783 main.py:320(pytest_collection)
1 0.000 0.000 26.783 26.783 main.py:567(perform_collect)
80557/556 0.020 0.000 26.108 0.047 {method 'extend' of 'list' objects}
85001/15001 0.151 0.000 26.103 0.002 main.py:785(genitems)
10002 0.047 0.000 25.324 0.003 runner.py:455(collect_one_node)
10002 0.045 0.000 24.888 0.002 runner.py:340(pytest_make_collect_report)
10002 0.069 0.000 24.805 0.002 runner.py:298(from_call)
10002 0.017 0.000 24.690 0.002 runner.py:341(<lambda>)
5001 0.168 0.000 23.150 0.005 python.py:412(collect)
5000 0.019 0.000 22.223 0.004 python.py:858(collect)
30003 0.101 0.000 19.818 0.001 python.py:218(pytest_pycollect_makeitem)
30000 0.161 0.000 19.368 0.001 python.py:450(_genfunctions)
30000 0.302 0.000 18.236 0.001 python.py:1611(from_parent)
40001 0.084 0.000 18.051 0.000 nodes.py:183(from_parent)
40002 0.116 0.000 17.967 0.000 nodes.py:102(_create)
30000 0.308 0.000 17.770 0.001 python.py:1549(__init__)
15000 0.117 0.000 16.111 0.001 fixtures.py:1439(getfixtureinfo)
15000 0.134 0.000 15.135 0.001 fixtures.py:1492(getfixtureclosure)
15000 9.320 0.001 14.738 0.001 fixtures.py:1479(_getautousenames)
--lf has an optimization where it skips collecting Modules (python
files) which don't contain failing tests. The optimization works by
getting the paths of all cached failed tests and skipping the collection
of Modules whose path is not included in that list.
In pytest, Package nodes are Module nodes with the fspath being the file
`<package dir>/__init__.py`. Since it's a Module the logic above
triggered for it, and because it's an `__init__.py` file which is
unlikely to have any failing tests in it, it is skipped, which causes
its entire directory to be skipped, including any Modules inside it with
failing tests.
Fix by special-casing Packages to never filter. This means entire
Packages are never filtered, the Modules themselves are always checked.
It is reasonable to consider an optimization which does filter entire
packages bases on parent paths etc. but this wouldn't actually save any
real work so is really not worth it.
Ran Benita
Simon K
Vasilis Gerakaris
Bruno Oliveira
Hugo van Kemenade
Tanvi Mehta
Tanvi Mehta
Kyle Altendorf
Anthony Sottile
Manuel Mariñez
Albert Villanova del Moral