add a README outlining plugin additions
This commit is contained in:
sommersoft
parent
a802ecc367
commit
1b947b269c
|
|
@ -0,0 +1,220 @@
|
|||
Automated Downstream Testing
|
||||
++++++++++++++++++++++++++++
|
||||
Testing pytest Plugins With Changes to pytest
|
||||
=============================================
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
- Utilizes Docker (compose) inside a GitHub Actions workflow to establish the necessary environment(s) to run tests.
|
||||
|
||||
- The base image is currently Ubuntu only, and uses ``pyenv`` to have the necessary versions of Python installed.
|
||||
|
||||
- The base image is on Docker Hub, which greatly speeds up each run by avoiding ``pyenv``'s compilation process each time.
|
||||
|
||||
- The completed image contains a PostgreSQL & mySQL layer (needed by ``pytest-django``, at minimum). Additional layers can be added if necessary.
|
||||
|
||||
- Utilizes each plugin's current test workflow to formulate the tests to run.
|
||||
|
||||
- The plugin's GitHub Actions workflow is used to get ``tox`` environment settings.
|
||||
|
||||
- The plugin's ``tox`` config is adjusted to ensure the local ``pytest`` is used.
|
||||
|
||||
- Uses a static command set: ``pip install tox`` & ``tox -e {toxenv}``.
|
||||
|
||||
How To Add Additional Plugin(s)
|
||||
-------------------------------
|
||||
**pytest/testing/downstream_testing/actions_schema.json:**
|
||||
**********************************************************
|
||||
|
||||
``actions_schema.json`` outlines both GitHub Actions workflow fields to get information, and how
|
||||
that information should be manipulated to provide usable tox environments to run a plugin's tests.
|
||||
|
||||
Example ``actions_schema.json`` entry:
|
||||
|
||||
.. code:: JSON
|
||||
|
||||
"pytest-django": {
|
||||
"matrix": [
|
||||
"matrix",
|
||||
"include"
|
||||
],
|
||||
"tox_cmd_build": {
|
||||
"base": "name",
|
||||
"prefix": "py",
|
||||
"sub": {
|
||||
"pattern": "-coverage$",
|
||||
"replace": ""
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Below are the steps to add a new plugin.
|
||||
|
||||
1. Start by adding a new JSON object, with the name of the repo. ``"pytest-django": {}`` in the example above.
|
||||
|
||||
2. Add an array named ``matrix``. Array items should point to the strategy matrix from the repo's GitHub Actions
|
||||
YAML file. For instance, the ``pytest-django`` example above has a strategy matrix defined as follows:
|
||||
|
||||
.. code:: YAML
|
||||
|
||||
# pytest-django/.github/workflows/main.yml
|
||||
jobs:
|
||||
tests:
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- name: linting,docs
|
||||
python: 3.8
|
||||
allow_failure: false
|
||||
|
||||
- name: py310-dj40-postgres-xdist-coverage
|
||||
python: '3.10'
|
||||
allow_failure: false
|
||||
|
||||
|
||||
This makes ``["matrix", "include"]`` our target to parse the ``pytest-django`` strategy. This is a "combination"
|
||||
strategy based on the use of ``include``.
|
||||
|
||||
For non-combination strategies, use the matrix field that points to the appropriate choices. Using
|
||||
``pytest-order`` as a non-combination example:
|
||||
|
||||
.. code:: YAML
|
||||
|
||||
# pytest-order/.github/workflows/pythontests.yml
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ${{ matrix.os }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
os: [ubuntu-latest, windows-latest]
|
||||
python-version: [3.6, 3.7, 3.8, 3.9, "3.10", pypy3]
|
||||
|
||||
The corresponding entry in ``actions_schema.json`` points to ``["matrix", "python-version"]``:
|
||||
|
||||
.. code:: JSON
|
||||
|
||||
"pytest-order": {
|
||||
"matrix": [
|
||||
"matrix",
|
||||
"python-version"
|
||||
],
|
||||
|
||||
3. Add a JSON object named ``tox_cmd_build``, with three items: ``base``, ``prefix``, and ``sub``.
|
||||
|
||||
- ``base``: For combination strategies (with ``include``), ``base`` is the field to be used as the basis
|
||||
of the tox environment. For non-combination strategies, this field is an empty string. ``base: "name"``
|
||||
in the ``pytest-django`` example above.
|
||||
|
||||
- ``prefix``: For combination strategies, ``prefix`` is used to [dis]qualify entries in ``base``. For
|
||||
non-combination strategies, this field is an emtpy string. ``prefix: "py"`` in the ``pytest-django``
|
||||
example above.
|
||||
|
||||
- ``sub``: For both combination and non-combination strategies, this JSON object gives a RegEx matching
|
||||
(``pattern``) and a substituition (``replace``) string. Since these are JSON strings, they cannot be
|
||||
represented as a Python raw string (``r""``); ensure to properly escape characters.
|
||||
|
||||
Any additions can be verified locally with the following process:
|
||||
|
||||
1. Have a local copy of a plugin's GitHub Actions workflow YAML file, as well as a ``tox.ini``. These should
|
||||
be placed in a folder; recommended folder name is the repo name.
|
||||
|
||||
.. code:: shell
|
||||
|
||||
.
|
||||
.. pytest-order/
|
||||
.... main.yml
|
||||
.... tox.ini
|
||||
|
||||
2. Utilize the ``--dry-run`` flag to run the parsing and command step building, without actually executing
|
||||
the commands. This will provide debugging information.
|
||||
|
||||
.. code::
|
||||
|
||||
(.venv) ~/pytest:$> python -m testing.downstream_testing.downstream_runner pytest-order pytest-order/pytest-order.yml test --matrix-exclude 3.6 --dry-run
|
||||
|
||||
DEBUG | downstream_runner.load_matrix_schema | Loading schema: /home/sommersoft/Dev/pytest-dev/pytest/testing/downstream_testing/action_schemas.json
|
||||
DEBUG | downstream_runner.load_matrix_schema | 'pytest-order' schema loaded: {'matrix': ['matrix', 'python-version'],
|
||||
'tox_cmd_build': {'base': '', 'prefix': '', 'sub': {'pattern': '(\\d|py\\d)\\.*(\\d+)', 'replace': 'py\\1\\2'}}, 'python_version': 'python-version'}
|
||||
DEBUG | downstream_runner.inject_pytest_dep | toxenv dependencies updated: {'!pytest{60,61,62,624,70}: pytest-xdist', '!pytest50: pytest @ file:///home/pytest'}
|
||||
DEBUG | downstream_runner.build_run | job_name: test
|
||||
DEBUG | downstream_runner.parse_matrix | parsed_matrix: [3.6, 3.7, 3.8, 3.9, '3.10', 'pypy3']
|
||||
DEBUG | downstream_runner.matrix | matrix: {'test': [{'name': 'py37', 'tox_cmd': 'py37'}, {'name': 'py38', 'tox_cmd': 'py38'}, {'name': 'py39', 'tox_cmd': 'py39'},
|
||||
{'name': 'py310', 'tox_cmd': 'py310'}, {'name': 'pypy3', 'tox_cmd': 'pypy3'}]}
|
||||
DEBUG | downstream_runner.build_run | matrix[job]: {'name': 'py37', 'tox_cmd': 'py37'}
|
||||
DEBUG | downstream_runner.build_run | matrix[job]: {'name': 'py38', 'tox_cmd': 'py38'}
|
||||
DEBUG | downstream_runner.build_run | matrix[job]: {'name': 'py39', 'tox_cmd': 'py39'}
|
||||
DEBUG | downstream_runner.build_run | matrix[job]: {'name': 'py310', 'tox_cmd': 'py310'}
|
||||
DEBUG | downstream_runner.build_run | matrix[job]: {'name': 'pypy3', 'tox_cmd': 'pypy3'}
|
||||
DEBUG | downstream_runner.build_run | built run: {'py37': ['pip install tox', 'tox -e py37'], 'py38': ['pip install tox', 'tox -e py38'], 'py39': ['pip install tox',
|
||||
'tox -e py39'], 'py310': ['pip install tox', 'tox -e py310'], 'pypy3': ['pip install tox', 'tox -e pypy3']}
|
||||
INFO | downstream_runner.run | --> running: 'pip install tox'
|
||||
INFO | downstream_runner.run | --> running: 'tox -e py37'
|
||||
INFO | downstream_runner.run | --> running: 'pip install tox'
|
||||
INFO | downstream_runner.run | --> running: 'tox -e py38'
|
||||
INFO | downstream_runner.run | --> running: 'pip install tox'
|
||||
INFO | downstream_runner.run | --> running: 'tox -e py39'
|
||||
INFO | downstream_runner.run | --> running: 'pip install tox'
|
||||
INFO | downstream_runner.run | --> running: 'tox -e py310'
|
||||
INFO | downstream_runner.run | --> running: 'pip install tox'
|
||||
INFO | downstream_runner.run | --> running: 'tox -e pypy3'
|
||||
|
||||
|
||||
**pytest/.github/workflows/downstream_testing.yml:**
|
||||
*****************************************************
|
||||
This GitHub Actions workflow orchestrates the various plugin tests, and only requires some minimal information.
|
||||
|
||||
Add a new entry to the combination strategy matrix:
|
||||
|
||||
1. ``name``: the plugin's repository name.
|
||||
|
||||
2. ``repo``: the ``org/name`` of the plugin's GitHub repository.
|
||||
|
||||
3. ``docker_profile``: the docker-compose profile to use for this entry. Docker profiles are defined in
|
||||
``pytest/testing/downstream_testing/docker-compose.yml``. At the time of this writing, the available
|
||||
profiles are: ``nodb``, ``mysql``, & ``postgres``.
|
||||
|
||||
4. ``jobs``: the list of job(s) to use from the plugin's GitHub Actions workflow.
|
||||
|
||||
5. ``workflow_name``: the name of the GitHub Actions workflow file to use (located in ``plugin/.github/workflows``).
|
||||
|
||||
6. ``matrix_exclude``: any entries to exclude from the selected matrix (combination & non-combination).
|
||||
|
||||
.. code:: YAML
|
||||
|
||||
- name: "pytest-django"
|
||||
repo: "pytest-dev/pytest-django"
|
||||
docker_profile: "postgres"
|
||||
jobs: "test"
|
||||
workflow_name: "main.yml"
|
||||
matrix_exclude: |
|
||||
linting,docs py39-dj40-mysql_innodb-coverage ...
|
||||
|
||||
- name: "pytest-django"
|
||||
repo: "pytest-dev/pytest-django"
|
||||
docker_profile: "mysql"
|
||||
jobs: "test"
|
||||
workflow_name: "main.yml"
|
||||
matrix_exclude: |
|
||||
linting,docs py310-dj40-postgres-xdist-coverage ...
|
||||
|
||||
.. epigraph::
|
||||
Example 1: using ``pytest-django``, which has a combination strategy matrix, we see two (of three) different
|
||||
``downstream_testing.yml`` entries. They each require a different database, so each entry uses the appropriate
|
||||
``docker_profile``. Additionally, to avoid known failures, ``matrix_exclude`` has all items that don't match
|
||||
the database in use.
|
||||
|
||||
.. code:: YAML
|
||||
|
||||
- name: "pytest-order"
|
||||
repo: "pytest-dev/pytest-order"
|
||||
docker_profile: "nodb"
|
||||
jobs: "test"
|
||||
workflow_name: "pythontests.yml"
|
||||
matrix_exclude: |
|
||||
3.6
|
||||
|
||||
.. epigraph::
|
||||
Example 2: using ``pytest-order``, which has a non-combination strategy matrix and requires no database.
|
||||
Loading…
Reference in New Issue