diff --git a/ISSUES.txt b/ISSUES.txt index d3900e24d..365d6fb5e 100644 --- a/ISSUES.txt +++ b/ISSUES.txt @@ -10,28 +10,27 @@ refine parametrize API ------------------------------------------------------------- tags: critical feature -extend metafunc.parametrize to better support indirection -by specifying a setupfunc(request, val) which will _substitute_ -the funcarg factory. Here is an example: +extend metafunc.parametrize to directly support indirection, example: - def setupdb(request, val): + def setupdb(request, config): # setup "resource" based on test request and the values passed # in to parametrize. setupfunc is called for each such value. # you may use request.addfinalizer() or request.cached_setup ... - return db + return dynamic_setup_database(val) @pytest.mark.parametrize("db", ["pg", "mysql"], setupfunc=setupdb) def test_heavy_functional_test(db): ... -There would be no need to write or explain funcarg factories here. +There would be no need to write or explain funcarg factories and +their special __ syntax. The examples and improvements should also show how to put the parametrize decorator to a class, to a module or even to a directory. For the directory part a conftest.py content like this:: pytestmark = [ - @pytest.mark.parametrize("db", ...), + @pytest.mark.parametrize_setup("db", ...), ] probably makes sense in order to keep the declarative nature. This mirrors @@ -39,13 +38,87 @@ the marker-mechanism with respect to a test module but puts it to a directory scale. When doing larger scoped parametrization it probably becomes neccessary -to allow parametrization to not match - currently any parametrized argument -that is not present in a function will cause a ValueError. With +to allow parametrization to be ignored if the according parameter is not +used (currently any parametrized argument that is not present in a function will cause a ValueError). Example: @pytest.mark.parametrize("db", ..., mustmatch=False) -we can tell pytest to not raise an error but simply ignore the parametrization -if the signature of a decorated function does not match. +means to not raise an error but simply ignore the parametrization +if the signature of a decorated function does not match. XXX is it +not sufficient to always allow non-matches? + + +unify item/request classes, generalize items +--------------------------------------------------------------- +tags: 2.4 wish + +in lieu of extended parametrization and the new way to specify resource +factories in terms of the parametrize decorator, consider unification +of the item and request class. This also is connected with allowing +funcargs in setup functions. Example of new item API: + + item.getresource("db") # alias for request.getfuncargvalue + item.addfinalizer(...) + item.cached_setup(...) + item.applymarker(...) + +test classes/modules could then use this api via:: + + def pytest_runtest_setup(item): + use item API ... + +introduction of this new method needs to be _fully_ backward compatible - +and the documentation needs to change along to mention this new way of +doing things. + +impl note: probably Request._fillfuncargs would be called from the +python plugins own pytest_runtest_setup(item) and would call +item.getresource(X) for all X in the funcargs of a function. + +XXX is it possible to even put the above item API to Nodes, i.e. also +to Directorty/module/file/class collectors? Problem is that current +funcarg factories presume they are called with a per-function (even +per-funcarg-per-function) scope. Could there be small tweaks to the new +API that lift this restriction? + +consider:: + + def setup_class(cls, tmpdir): + # would get a per-class tmpdir because tmpdir parametrization + # would know that it is called with a class scope + # + # + # +this looks very difficult because those setup functions are also used +by nose etc. Rather consider introduction of a new setup hook: + + def setup_test(self, item): + self.db = item.cached_setup(..., scope='class') + self.tmpdir = item.getresource("tmpdir") + +this should be compatible to unittest/nose and provide much of what +"testresources" provide. XXX This would not allow full parametrization +such that test function could be run multiple times with different +values. See "parametrized attributes" issue. + +allow parametrized attributes on classes +-------------------------------------------------- + +tags: wish 2.4 + +example: + + @pytest.mark.parametrize_attr("db", setupfunc, [1,2,3], scope="class") + @pytest.mark.parametrize_attr("tmp", setupfunc, scope="...") + class TestMe: + def test_hello(self): + access self.db ... + +this would run the test_hello() function three times with three +different values for self.db. This could also work with unittest/nose +style tests, i.e. it leverages existing test suites without needing +to rewrite them. Together with the previously mentioned setup_test() +maybe the setupfunc could be ommitted? checks / deprecations for next release --------------------------------------------------------------- @@ -74,18 +147,6 @@ appropriately to avoid this issue. Moreover/Alternatively, we could record which implementations of a hook succeeded and only call their teardown. -do early-teardown of test modules ------------------------------------------ -tags: feature - -currently teardowns are called when the next tests is setup -except for the function/method level where interally -"teardown_exact" tears down immediately. Generalize -this to perform the "neccessary" teardown compared to -the "next" test item during teardown - this should -get rid of some irritations because otherwise e.g. -prints of teardown-code appear in the setup of the next test. - consider and document __init__ file usage in test directories --------------------------------------------------------------- tags: bug core @@ -167,6 +228,7 @@ attempt will be made). consider introducing a hook/mechanism that allows to apply marks from conftests or plugins. (See extended parametrization) + explicit referencing of conftest.py files ----------------------------------------- tags: feature