It should happen as late as possible before the test runs. Ref: https://github.com/pytest-dev/pytest/issues/7022