diff --git a/testing/downstream_testing/README.rst b/testing/downstream_testing/README.rst new file mode 100644 index 000000000..025a05649 --- /dev/null +++ b/testing/downstream_testing/README.rst @@ -0,0 +1,220 @@ +Automated Downstream Testing +++++++++++++++++++++++++++++ +Testing pytest Plugins With Changes to pytest +============================================= + +Overview +-------- + +- Utilizes Docker (compose) inside a GitHub Actions workflow to establish the necessary environment(s) to run tests. + + - The base image is currently Ubuntu only, and uses ``pyenv`` to have the necessary versions of Python installed. + + - The base image is on Docker Hub, which greatly speeds up each run by avoiding ``pyenv``'s compilation process each time. + + - The completed image contains a PostgreSQL & mySQL layer (needed by ``pytest-django``, at minimum). Additional layers can be added if necessary. + +- Utilizes each plugin's current test workflow to formulate the tests to run. + + - The plugin's GitHub Actions workflow is used to get ``tox`` environment settings. + + - The plugin's ``tox`` config is adjusted to ensure the local ``pytest`` is used. + +- Uses a static command set: ``pip install tox`` & ``tox -e {toxenv}``. + +How To Add Additional Plugin(s) +------------------------------- +**pytest/testing/downstream_testing/actions_schema.json:** +********************************************************** + +``actions_schema.json`` outlines both GitHub Actions workflow fields to get information, and how +that information should be manipulated to provide usable tox environments to run a plugin's tests. + +Example ``actions_schema.json`` entry: + + .. code:: JSON + + "pytest-django": { + "matrix": [ + "matrix", + "include" + ], + "tox_cmd_build": { + "base": "name", + "prefix": "py", + "sub": { + "pattern": "-coverage$", + "replace": "" + } + } + } + +Below are the steps to add a new plugin. + +1. Start by adding a new JSON object, with the name of the repo. ``"pytest-django": {}`` in the example above. + +2. Add an array named ``matrix``. Array items should point to the strategy matrix from the repo's GitHub Actions + YAML file. For instance, the ``pytest-django`` example above has a strategy matrix defined as follows: + + .. code:: YAML + + # pytest-django/.github/workflows/main.yml + jobs: + tests: + strategy: + fail-fast: false + matrix: + include: + - name: linting,docs + python: 3.8 + allow_failure: false + + - name: py310-dj40-postgres-xdist-coverage + python: '3.10' + allow_failure: false + + + This makes ``["matrix", "include"]`` our target to parse the ``pytest-django`` strategy. This is a "combination" + strategy based on the use of ``include``. + + For non-combination strategies, use the matrix field that points to the appropriate choices. Using + ``pytest-order`` as a non-combination example: + + .. code:: YAML + + # pytest-order/.github/workflows/pythontests.yml + jobs: + test: + runs-on: ${{ matrix.os }} + strategy: + fail-fast: false + matrix: + os: [ubuntu-latest, windows-latest] + python-version: [3.6, 3.7, 3.8, 3.9, "3.10", pypy3] + + The corresponding entry in ``actions_schema.json`` points to ``["matrix", "python-version"]``: + + .. code:: JSON + + "pytest-order": { + "matrix": [ + "matrix", + "python-version" + ], + +3. Add a JSON object named ``tox_cmd_build``, with three items: ``base``, ``prefix``, and ``sub``. + + - ``base``: For combination strategies (with ``include``), ``base`` is the field to be used as the basis + of the tox environment. For non-combination strategies, this field is an empty string. ``base: "name"`` + in the ``pytest-django`` example above. + + - ``prefix``: For combination strategies, ``prefix`` is used to [dis]qualify entries in ``base``. For + non-combination strategies, this field is an emtpy string. ``prefix: "py"`` in the ``pytest-django`` + example above. + + - ``sub``: For both combination and non-combination strategies, this JSON object gives a RegEx matching + (``pattern``) and a substituition (``replace``) string. Since these are JSON strings, they cannot be + represented as a Python raw string (``r""``); ensure to properly escape characters. + +Any additions can be verified locally with the following process: + +1. Have a local copy of a plugin's GitHub Actions workflow YAML file, as well as a ``tox.ini``. These should + be placed in a folder; recommended folder name is the repo name. + + .. code:: shell + + . + .. pytest-order/ + .... main.yml + .... tox.ini + +2. Utilize the ``--dry-run`` flag to run the parsing and command step building, without actually executing + the commands. This will provide debugging information. + + .. code:: + + (.venv) ~/pytest:$> python -m testing.downstream_testing.downstream_runner pytest-order pytest-order/pytest-order.yml test --matrix-exclude 3.6 --dry-run + + DEBUG | downstream_runner.load_matrix_schema | Loading schema: /home/sommersoft/Dev/pytest-dev/pytest/testing/downstream_testing/action_schemas.json + DEBUG | downstream_runner.load_matrix_schema | 'pytest-order' schema loaded: {'matrix': ['matrix', 'python-version'], + 'tox_cmd_build': {'base': '', 'prefix': '', 'sub': {'pattern': '(\\d|py\\d)\\.*(\\d+)', 'replace': 'py\\1\\2'}}, 'python_version': 'python-version'} + DEBUG | downstream_runner.inject_pytest_dep | toxenv dependencies updated: {'!pytest{60,61,62,624,70}: pytest-xdist', '!pytest50: pytest @ file:///home/pytest'} + DEBUG | downstream_runner.build_run | job_name: test + DEBUG | downstream_runner.parse_matrix | parsed_matrix: [3.6, 3.7, 3.8, 3.9, '3.10', 'pypy3'] + DEBUG | downstream_runner.matrix | matrix: {'test': [{'name': 'py37', 'tox_cmd': 'py37'}, {'name': 'py38', 'tox_cmd': 'py38'}, {'name': 'py39', 'tox_cmd': 'py39'}, + {'name': 'py310', 'tox_cmd': 'py310'}, {'name': 'pypy3', 'tox_cmd': 'pypy3'}]} + DEBUG | downstream_runner.build_run | matrix[job]: {'name': 'py37', 'tox_cmd': 'py37'} + DEBUG | downstream_runner.build_run | matrix[job]: {'name': 'py38', 'tox_cmd': 'py38'} + DEBUG | downstream_runner.build_run | matrix[job]: {'name': 'py39', 'tox_cmd': 'py39'} + DEBUG | downstream_runner.build_run | matrix[job]: {'name': 'py310', 'tox_cmd': 'py310'} + DEBUG | downstream_runner.build_run | matrix[job]: {'name': 'pypy3', 'tox_cmd': 'pypy3'} + DEBUG | downstream_runner.build_run | built run: {'py37': ['pip install tox', 'tox -e py37'], 'py38': ['pip install tox', 'tox -e py38'], 'py39': ['pip install tox', + 'tox -e py39'], 'py310': ['pip install tox', 'tox -e py310'], 'pypy3': ['pip install tox', 'tox -e pypy3']} + INFO | downstream_runner.run | --> running: 'pip install tox' + INFO | downstream_runner.run | --> running: 'tox -e py37' + INFO | downstream_runner.run | --> running: 'pip install tox' + INFO | downstream_runner.run | --> running: 'tox -e py38' + INFO | downstream_runner.run | --> running: 'pip install tox' + INFO | downstream_runner.run | --> running: 'tox -e py39' + INFO | downstream_runner.run | --> running: 'pip install tox' + INFO | downstream_runner.run | --> running: 'tox -e py310' + INFO | downstream_runner.run | --> running: 'pip install tox' + INFO | downstream_runner.run | --> running: 'tox -e pypy3' + + +**pytest/.github/workflows/downstream_testing.yml:** +***************************************************** +This GitHub Actions workflow orchestrates the various plugin tests, and only requires some minimal information. + +Add a new entry to the combination strategy matrix: + +1. ``name``: the plugin's repository name. + +2. ``repo``: the ``org/name`` of the plugin's GitHub repository. + +3. ``docker_profile``: the docker-compose profile to use for this entry. Docker profiles are defined in + ``pytest/testing/downstream_testing/docker-compose.yml``. At the time of this writing, the available + profiles are: ``nodb``, ``mysql``, & ``postgres``. + +4. ``jobs``: the list of job(s) to use from the plugin's GitHub Actions workflow. + +5. ``workflow_name``: the name of the GitHub Actions workflow file to use (located in ``plugin/.github/workflows``). + +6. ``matrix_exclude``: any entries to exclude from the selected matrix (combination & non-combination). + +.. code:: YAML + + - name: "pytest-django" + repo: "pytest-dev/pytest-django" + docker_profile: "postgres" + jobs: "test" + workflow_name: "main.yml" + matrix_exclude: | + linting,docs py39-dj40-mysql_innodb-coverage ... + + - name: "pytest-django" + repo: "pytest-dev/pytest-django" + docker_profile: "mysql" + jobs: "test" + workflow_name: "main.yml" + matrix_exclude: | + linting,docs py310-dj40-postgres-xdist-coverage ... + +.. epigraph:: + Example 1: using ``pytest-django``, which has a combination strategy matrix, we see two (of three) different + ``downstream_testing.yml`` entries. They each require a different database, so each entry uses the appropriate + ``docker_profile``. Additionally, to avoid known failures, ``matrix_exclude`` has all items that don't match + the database in use. + +.. code:: YAML + + - name: "pytest-order" + repo: "pytest-dev/pytest-order" + docker_profile: "nodb" + jobs: "test" + workflow_name: "pythontests.yml" + matrix_exclude: | + 3.6 + +.. epigraph:: + Example 2: using ``pytest-order``, which has a non-combination strategy matrix and requires no database.