Now `importlib` mode will correctly set the imported modules as an attribute of their parent modules.
As helpfully posted on #12194, that's how the Python import module works so we should follow suit.
In addition, we also try to import the parent modules as part of the process of importing a child module, again mirroring how Python importing works.
Fix#12194
Previously we used a hand crafted approach to detect namespace packages, however we should rely on ``importlib`` to detect them for us.
Fix#12112
---------
Co-authored-by: Ran Benita <ran@unusedvar.com>
As detailed in
https://github.com/pytest-dev/pytest/issues/11475#issuecomment-1937043670,
currently with `--import-mode=importlib` pytest will try to import every
file by using a unique module name, regardless if that module could be
imported using the normal import mechanism without touching `sys.path`.
This has the consequence that non-test modules available in `sys.path`
(via other mechanism, such as being installed into a virtualenv,
PYTHONPATH, etc) would end up being imported as standalone modules,
instead of imported with their expected module names.
To illustrate:
```
.env/
lib/
site-packages/
anndata/
core.py
```
Given `anndata` is installed into the virtual environment, `python -c
"import anndata.core"` works, but pytest with `importlib` mode would
import that module as a standalone module named
`".env.lib.site-packages.anndata.core"`, because importlib module was
designed to import test files which are not reachable from `sys.path`,
but now it is clear that normal modules should be imported using the
standard mechanisms if possible.
Now `imporlib` mode will first try to import the module normally,
without changing `sys.path`, and if that fails it falls back to
importing the module as a standalone module.
This also makes `importlib` respect namespace packages.
This supersedes #11931.
Fix#11475Close#11931
ruff is faster and handle everything we had prior.
isort configuration done based on the indication from
https://github.com/astral-sh/ruff/issues/4670, previousely based on
reorder-python-import (#11896)
flake8-docstrings was a wrapper around pydocstyle (now archived) that
explicitly asks to use ruff in https://github.com/PyCQA/pydocstyle/pull/658.
flake8-typing-import is useful mainly for project that support python 3.7
and the one useful check will be implemented in https://github.com/astral-sh/ruff/issues/2302
We need to keep blacken-doc because ruff does not handle detection
of python code inside .md and .rst. The direct link to the repo is
now used to avoid a redirection.
Manual fixes:
- Lines that became too long
- % formatting that was not done automatically
- type: ignore that were moved around
- noqa of hard to fix issues (UP031 generally)
- fmt: off and fmt: on that is not really identical
between black and ruff
- autofix re-order in pre-commit from faster to slower
Co-authored-by: Ran Benita <ran@unusedvar.com>
Change our mypy configuration to disallow untyped defs by default, which ensures *new* files added to the code base are fully typed.
To avoid having to type-annotate everything now, add `# mypy: allow-untyped-defs` to files which are not fully type annotated yet.
As we fully type annotate those modules, we can then just remove that directive from the top.
The function currently uses `find_suffixes` which iterates the entire
directory searching for files with the given suffix. In some cases
though, like in pytest's selftest, the directory can get big:
$ ls /tmp/pytest-of-ran/pytest-0/
7686
and iterating it many times can get slow.
This doesn't fix the underlying issue (iterating the directory) but at
least speeds it up a bit by using `os.scandir` instead of
`path.iterdir`. So `make_numbered_dir` is still slow for pytest's
selftests, but reduces ~10s for me.
Starting with `resolve_package_path` and its associated tests,
this refactoring seeks to make path concatenation more
readable and consistent within tests/functions.
As discussed in #11413:
- code is free to use either `/` and `joinpath`
- consistency within a function is more important than consistency across the codebase
- it is nice to use `/` when it is more readable
- it is nice to use `joinpath` when there is little context
- be mindful that `joinpath` may be clearer when joining multiple segments
For packages, `import_path` receives the path to the package's `__init__.py` file, however module names (as they live in `sys.modules`) should not include the `__init__` part.
For example, `app/core/__init__.py` should be imported as `app.core`, not as `app.core.__init__`.
Fix#11306
The initial implementation (in #7246) introduced the `importlib` mode, which
never added the imported module to `sys.modules`, so it included a test
to ensure calling `import_path` twice would yield different modules.
Not adding modules to `sys.modules` proved problematic, so we began to add the imported module to `sys.modules`
in #7870, but failed to realize that given we are now changing `sys.modules`, we might
as well avoid importing it more than once.
Then #10088 came along, passing `importlib` also when importing application modules
(as opposed to only test modules before), which caused problems due to imports
having side-effects and the expectation being that they are imported only once.
With this PR, `import_path` returns the module immediately if already in
`sys.modules`.
Fix#10811Fix#10341
The problem is that we would loop over all directories of the basetemp directory searching for dead symlinks, for each test, which would compound over the test session run.
Doing the cleanup just once, at the end of the session, fixes the problem.
Fix#10896
Nowadays, the prepend and append import modes use importlib.import_module() instead of __import__().
There was a phrase “which avoids having to use `__import__`”, in which I couldn’t just replace `__import__` by `importlib.import_module` because the latter is used (in insert_missing_modules()) also when using importlib mode. Therefore I removed the part from the sentence.
The dummy modules we introduce in `insert_missing_modules` (due to #7856 and #7859)
would cause problems if the dummy modules actually end up replacing modules
which could be imported normally because they are available in `PYTHONPATH`.
Now we attempt to first import the module via normal mechanisms, and only
introduce the dummy modules if the intermediary modules don't actually exist.
Close#9645
* [pre-commit.ci] pre-commit autoupdate
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Ran Benita <ran@unusedvar.com>
(Written for a Unix system, but might be applicable to Windows as well).
pytest creates a root temporary directory under /tmp, named
`pytest-of-<username>`, and creates tmp_path's and other under it.
/tmp is shared between all users of the system.
This root temporary directory was created with 0o777&~umask permissions,
which usually becomes 0o755, meaning any user in the system could list
and read the files, which is undesirable.
Use 0o700 permissions instead. Also for subdirectories, because the root
dir is adjustable.
On Windows, os.path.samefile returns false for paths mounted in UNC paths which
point to the same location.
I couldn't reproduce the actual case reported, but looking at the code it seems
this commit should fix the issue.
Fix#7678Fix#8076
When pytest was run on a directory containing a recursive symlink it failed
with ELOOP as the library was not able to determine the type of the
direntry:
src/_pytest/main.py:685: in collect
if not direntry.is_file():
E OSError: [Errno 40] Too many levels of symbolic links: '/home/florian/proj/pytest/tests/recursive'
This is fixed by handling ELOOP and other errors in the visit function in
pathlib.py, so the entries whose is_file() call raises an OSError with the
pre-defined list of error numbers will be exluded from the result.
The _ignore_errors function was copied from Lib/pathlib.py of cpython 3.9.
Fixes#7951
Josh Soref
Ran Benita
Bruno Oliveira
Bruno Oliveira
Pierre Sassoulas
Warren Markham
akhilramkee
Manuel Jacob
Daniel Scheffler
Yusuke Kadowaki
Anthony Sottile
pre-commit-ci[bot]
Harshna
Tadeu Manoel
Cserna Zsolt
Vasilis Gerakaris