update todo to make some sense again
shift samuele in license file --HG-- branch : trunk
This commit is contained in:
		
							parent
							
								
									a58292b473
								
							
						
					
					
						commit
						d4a28b1b2d
					
				
							
								
								
									
										222
									
								
								TODO.txt
								
								
								
								
							
							
						
						
									
										222
									
								
								TODO.txt
								
								
								
								
							| 
						 | 
					@ -4,111 +4,39 @@ Things to do for 1.0.0
 | 
				
			||||||
py.test 
 | 
					py.test 
 | 
				
			||||||
--------------
 | 
					--------------
 | 
				
			||||||
 | 
					
 | 
				
			||||||
- compatilibity: honour/warn item.run() method of test items (so far
 | 
					- clarify setup/run events and runner.py versus pytest_runner.py, 
 | 
				
			||||||
  probably only execute() is warned about or the other way round)
 | 
					  introduce a general pytest_item_setup(item, setupstate)
 | 
				
			||||||
 | 
					  and always isolate py._com.comregistry when py lib's own tests are run
 | 
				
			||||||
 | 
					
 | 
				
			||||||
- introduce plugin arch, port existing things to plugins:
 | 
					- hook review and hook docs 
 | 
				
			||||||
  - importorskip 
 | 
					 | 
				
			||||||
  - filelog 
 | 
					 | 
				
			||||||
  - chtmpdir per method 
 | 
					 | 
				
			||||||
  - apigen 
 | 
					 | 
				
			||||||
  - xfail 
 | 
					 | 
				
			||||||
  - acceptance/assertlines 
 | 
					 | 
				
			||||||
  - dist-testing? 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
- introduce setuptools-style version checking, at least
 | 
					 | 
				
			||||||
  for py lib itself, maybe also for other packages: 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
     py.checkversion("py>=1.0")
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
- generative tests: it is somewhat misleading for 
 | 
					 | 
				
			||||||
  the classical htmlviews that generated tests are identified 
 | 
					 | 
				
			||||||
  by numbers rather than by its parameters.  unclear how
 | 
					 | 
				
			||||||
  to fix this because we may not always be able to assume
 | 
					 | 
				
			||||||
  that we can identify a generated tests by using its parameters
 | 
					 | 
				
			||||||
  (it might not be hashable, doesn't have a sensical repr ...?)
 | 
					 | 
				
			||||||
 | 
					
 | 
				
			||||||
- turn deprecation / apiwarnings into events, report them at the end? 
 | 
					- turn deprecation / apiwarnings into events, report them at the end? 
 | 
				
			||||||
 | 
					
 | 
				
			||||||
- get APIGEN back to work 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
- get web reporter back to work 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
- introduce decorator "shouldfail" or "xfail" 
 | 
					 | 
				
			||||||
  as to mark a test as "expected to fail", 
 | 
					 | 
				
			||||||
  report specially if it surprisingly passes 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
- nightly test runs on multiple platforms 
 | 
					- nightly test runs on multiple platforms 
 | 
				
			||||||
 | 
					
 | 
				
			||||||
- review and refactor architecture of py.test with particular
 | 
					 | 
				
			||||||
  respect to: 
 | 
					 | 
				
			||||||
  - writing (stacked) extensions / plugins (compared to Nose) 
 | 
					 | 
				
			||||||
  - porting existing extensions (htmlconftest / buildbot / PyPy's conftest's ...) 
 | 
					 | 
				
			||||||
  - fast and stable distributed testing 
 | 
					 | 
				
			||||||
  - reliable cross-platform testing 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
- improve py.test documentation to reflect new
 | 
					 | 
				
			||||||
  event architecture 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
- review and optimize skip-handling (it can be quite slow in
 | 
					 | 
				
			||||||
  certain situations because e.g. setup/teardown is fully performed 
 | 
					 | 
				
			||||||
  although we have "skip by keyword" and could detect this early)
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
py.execnet
 | 
					py.execnet
 | 
				
			||||||
--------------
 | 
					--------------
 | 
				
			||||||
 | 
					
 | 
				
			||||||
- cross-python version (2.2/2.3-2.5/6) and cross-platform testing of 
 | 
					- cross-python version (2.2/2.3-2.5/6) and cross-platform testing of 
 | 
				
			||||||
  setup/teardown semantics
 | 
					  setup/teardown semantics
 | 
				
			||||||
 | 
					
 | 
				
			||||||
- optimize general setup and rsync timing? 
 | 
					py.test apigen plugin
 | 
				
			||||||
 | 
					---------------------------
 | 
				
			||||||
 | 
					
 | 
				
			||||||
py.apigen 
 | 
					- make it work again with the new plugin arch
 | 
				
			||||||
----------------
 | 
					 | 
				
			||||||
 | 
					
 | 
				
			||||||
- make it work again 
 | 
					packaging / svn-mercurial interaction
 | 
				
			||||||
 | 
					--------------------------------------------
 | 
				
			||||||
 | 
					
 | 
				
			||||||
see apigen_refactorings.txt
 | 
					- decide if to go with or without setuptools, check windows 2.6
 | 
				
			||||||
 | 
					  availability 
 | 
				
			||||||
 | 
					
 | 
				
			||||||
- check out CodeInvestigator 
 | 
					- open a mercurial branch for releases? 
 | 
				
			||||||
  http://codeinvestigator.googlepages.com/main
 | 
					 | 
				
			||||||
 | 
					
 | 
				
			||||||
  or other code that collects data from running a program 
 | 
					- write a script to dumb-bridge the mercurial repo to svn 
 | 
				
			||||||
  (in our case running the tests)
 | 
					  (i.e. forget about svn history)
 | 
				
			||||||
 | 
					
 | 
				
			||||||
Criticism and solutions 
 | 
					1.1 and beyond 
 | 
				
			||||||
--------------------------------
 | 
					 | 
				
			||||||
"too big": 
 | 
					 | 
				
			||||||
    - too much code, you need entire py lib, hard to include into app 
 | 
					 | 
				
			||||||
      + have a small pytest boostrap that loads pylib.zip from net
 | 
					 | 
				
			||||||
      + provide smaller script ala simpy 
 | 
					 | 
				
			||||||
    - lots of cmdline options, possibilities, documentation
 | 
					 | 
				
			||||||
      rather unsorted 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
"needless differences between py.test and nosetests":
 | 
					 | 
				
			||||||
    - py.test.skip 
 | 
					 | 
				
			||||||
    - raises  
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
"tutorial structure missing", e.g.:
 | 
					 | 
				
			||||||
    - "how to get started" in a minimal way, also how to use
 | 
					 | 
				
			||||||
      existing conftests/plugins 
 | 
					 | 
				
			||||||
    - how to configure py.test 
 | 
					 | 
				
			||||||
    - how to write plugins/extensions 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
"too much magic"
 | 
					 | 
				
			||||||
   - re-execution of assert expressions 
 | 
					 | 
				
			||||||
     + rename "--nomagic" to something that turns off "superassertions" 
 | 
					 | 
				
			||||||
     + hint at --tb=... 
 | 
					 | 
				
			||||||
   - get rid of py/magic directory 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
has a good ui but could be better
 | 
					 | 
				
			||||||
    - support developer communication, e.g. py.test
 | 
					 | 
				
			||||||
      --sendfailures=freenode-pypy 
 | 
					 | 
				
			||||||
      --sendfailures=pocoo # prints out paste.pocoo.url with traceback 
 | 
					 | 
				
			||||||
    - graphical interface, probably QT 
 | 
					 | 
				
			||||||
    - generally store test results and use them for subsequent calls 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
ld (review and shift to above)
 | 
					 | 
				
			||||||
=================================
 | 
					=================================
 | 
				
			||||||
 | 
					
 | 
				
			||||||
refactorings
 | 
					refactorings
 | 
				
			||||||
| 
						 | 
					@ -119,16 +47,6 @@ refactorings
 | 
				
			||||||
  - check if it works on win32 
 | 
					  - check if it works on win32 
 | 
				
			||||||
  - refine error reporting (don't show python tracebacks) 
 | 
					  - refine error reporting (don't show python tracebacks) 
 | 
				
			||||||
 | 
					
 | 
				
			||||||
- generalization of "host specifications" for execnet and
 | 
					 | 
				
			||||||
  py.test --dist usages in particular (see also revision 37500 which 
 | 
					 | 
				
			||||||
  contained a draft for that).  The goal is to have cross-platform 
 | 
					 | 
				
			||||||
  testing and dist-testing and other usages of py.execnet all 
 | 
					 | 
				
			||||||
  use a common syntax for specifiying connection methods and 
 | 
					 | 
				
			||||||
  be able to instantiate gateways/connections through it. 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
- unification of "gateway"/host setup and teardown, including
 | 
					 | 
				
			||||||
  rsyncing, i.e. cross-platform and dist-testing. 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
- py.log: unify API, possibly deprecate duplicate ones, 
 | 
					- py.log: unify API, possibly deprecate duplicate ones, 
 | 
				
			||||||
  base things on a Config object  (hte latter almost a feature though) 
 | 
					  base things on a Config object  (hte latter almost a feature though) 
 | 
				
			||||||
  (M988)
 | 
					  (M988)
 | 
				
			||||||
| 
						 | 
					@ -139,7 +57,9 @@ features
 | 
				
			||||||
--------------
 | 
					--------------
 | 
				
			||||||
 | 
					
 | 
				
			||||||
- (Harald Armin Massa): make py2exe work with py lib 
 | 
					- (Harald Armin Massa): make py2exe work with py lib 
 | 
				
			||||||
 | 
					
 | 
				
			||||||
- optimize file checking with --looponfailing (harald has code for win32)
 | 
					- optimize file checking with --looponfailing (harald has code for win32)
 | 
				
			||||||
 | 
					
 | 
				
			||||||
- have a py.test scan/run database for results and test names
 | 
					- have a py.test scan/run database for results and test names
 | 
				
			||||||
  etc. (to allow quicker selection of tests and post-run 
 | 
					  etc. (to allow quicker selection of tests and post-run 
 | 
				
			||||||
  information on failures etc.)  (M760) 
 | 
					  information on failures etc.)  (M760) 
 | 
				
			||||||
| 
						 | 
					@ -168,116 +88,8 @@ features
 | 
				
			||||||
- references from ReST docs to modules, functions and classes
 | 
					- references from ReST docs to modules, functions and classes
 | 
				
			||||||
  of apigen generated html docs (M960)
 | 
					  of apigen generated html docs (M960)
 | 
				
			||||||
 | 
					
 | 
				
			||||||
- review svn-testing (and escape characters), consider
 | 
					 | 
				
			||||||
  svn-bindings (M634)
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
- py.test.pdb - there is my hack for a while now, which integrates
 | 
					- py.test.pdb - there is my hack for a while now, which integrates
 | 
				
			||||||
  rlcompleter2 with pdb. First of all it requires some strange changes
 | 
					  rlcompleter2 with pdb. First of all it requires some strange changes
 | 
				
			||||||
  to rlcompleter itself, which has no tests. Long-term plan would be
 | 
					  to rlcompleter itself, which has no tests. Long-term plan would be
 | 
				
			||||||
  to have pyrepl+rlcompleter2+pdb fixes integrated into pylib and
 | 
					  to have pyrepl+rlcompleter2+pdb fixes integrated into pylib and
 | 
				
			||||||
  have it tested. This requires work though.
 | 
					  have it tested. This requires work though.
 | 
				
			||||||
 | 
					 | 
				
			||||||
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 | 
					 | 
				
			||||||
--- below neeeds more review ---
 | 
					 | 
				
			||||||
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
More random notes, goals
 | 
					 | 
				
			||||||
--------------------------------
 | 
					 | 
				
			||||||
- REDUCE "MAGICALNESS", from an IRC discussion with ronny:
 | 
					 | 
				
			||||||
  - integration into IDEs
 | 
					 | 
				
			||||||
  - python2.6/python3 compat? 
 | 
					 | 
				
			||||||
  - initpkg exports
 | 
					 | 
				
			||||||
  - assert reinterpretation 
 | 
					 | 
				
			||||||
  - greenlet compiles at runtime only in "dev-mode"
 | 
					 | 
				
			||||||
  - conftest's are "scary", hum, what about 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
nicefications: 
 | 
					 | 
				
			||||||
looponfailing shoudl nicely signal tests that failed but now PASS tests
 | 
					 | 
				
			||||||
rename Node to itemtestloop
 | 
					 | 
				
			||||||
refactor config and session tests to go into test_distsession.py 
 | 
					 | 
				
			||||||
raises DID NOT RAISE: report the return value
 | 
					 | 
				
			||||||
have node.shutdown perform out-of-band so that shutdowns happens more quickly
 | 
					 | 
				
			||||||
tracebacks of importerrors of test modules should start with the test module file
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
- fix hostmanage to care for setting PYTHONPATH properly
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
- BRANCH: 
 | 
					 | 
				
			||||||
  adding of options 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
- BRANCH: 
 | 
					 | 
				
			||||||
    TEMPDIR handling 
 | 
					 | 
				
			||||||
    syspath handling (notify on changes, restore for each test?) 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
- COMPLETE REPORTING FOR MERGE!
 | 
					 | 
				
			||||||
  - remove ItemStart and CollectionStart 
 | 
					 | 
				
			||||||
    which are only needed for collectonly. 
 | 
					 | 
				
			||||||
    implement it some other way. 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
  - pre-counting of test items 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
  - move assert reinterp back to session? 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
  - merge "--tb" and "--fulltrace" option, --tb=full 
 | 
					 | 
				
			||||||
  - implement --showouterr, don't show outerr by default 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
  - domainpath? 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
  - translate remote filenames to local filenames so that,
 | 
					 | 
				
			||||||
    probably based on option
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
  - review safe_repr
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
  - move OutcomeRepr.where/exconly attr to ReprExceptionInfo 
 | 
					 | 
				
			||||||
    or substitute as LocationRepr? 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
- reprcrash rename message to exconly 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
- test terminal reporter 
 | 
					 | 
				
			||||||
  ACCEPTANCE test for eventlog writing
 | 
					 | 
				
			||||||
  ACCEPTANCE test for acceptance custom reporting :) 
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
  ACCEPTANCE showing nice Collection Errors  
 | 
					 | 
				
			||||||
    
 | 
					 | 
				
			||||||
  ACCEPTANCE test for "py.test2" exit signals 
 | 
					 | 
				
			||||||
  ACCEPTANCE test for "py.test2 --traceconfig" 
 | 
					 | 
				
			||||||
  ACCEPTANCE test for nice reprsentation of failures during Collection
 | 
					 | 
				
			||||||
  ACCEPTANCE test for nice reprsentation of failures during Generator Collection 
 | 
					 | 
				
			||||||
  ACCEPTANCE test for "py.test2" conftest containing syntax errors
 | 
					 | 
				
			||||||
  
 | 
					 | 
				
			||||||
  ACCEPTANCE test for "py.test2" honouring conftest specifying "extrainfo"
 | 
					 | 
				
			||||||
  ACCEPTANCE test for "py.test2" on a simple example project 
 | 
					 | 
				
			||||||
 
 | 
					 | 
				
			||||||
- merge CollectionFinish and ItemTestReport
 | 
					 | 
				
			||||||
  maybe with base class: BaseReport and some common attrs/methods
 | 
					 | 
				
			||||||
 
 | 
					 | 
				
			||||||
- merge terminal/remote and dist-testing
 | 
					 | 
				
			||||||
  and make allocation of tests to hosts more dynamic
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
- time setup/teardown and the actual test runs separately 
 | 
					 | 
				
			||||||
  ACCEPTANCE:
 | 
					 | 
				
			||||||
  py.test2 --
 | 
					 | 
				
			||||||
  py.test2 --repeat=10
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
- expect failing tests 
 | 
					 | 
				
			||||||
  py.test2.expectfail(feature=138)
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
while killing a process:
 | 
					 | 
				
			||||||
Exception in thread receiver:
 | 
					 | 
				
			||||||
Traceback (most recent call last):
 | 
					 | 
				
			||||||
  File "threading.py", line 460, in __bootstrap
 | 
					 | 
				
			||||||
    self.run()
 | 
					 | 
				
			||||||
  File "threading.py", line 440, in run
 | 
					 | 
				
			||||||
    self.__target(*self.__args, **self.__kwargs)
 | 
					 | 
				
			||||||
  File "/home/hpk/py/branch/event/py/execnet/gateway.py", line 140, in _thread_receiver
 | 
					 | 
				
			||||||
    self._stopsend()
 | 
					 | 
				
			||||||
  File "/home/hpk/py/branch/event/py/execnet/gateway.py", line 329, in _stopsend
 | 
					 | 
				
			||||||
    self._send(None)
 | 
					 | 
				
			||||||
  File "/home/hpk/py/branch/event/py/execnet/gateway.py", line 147, in _send
 | 
					 | 
				
			||||||
    self._io.close_write()
 | 
					 | 
				
			||||||
  File "/home/hpk/py/branch/event/py/execnet/inputoutput.py", line 106, in close_write
 | 
					 | 
				
			||||||
    self.outfile.close()
 | 
					 | 
				
			||||||
IOError: [Errno 32] Broken pipe
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
 | 
					 | 
				
			||||||
| 
						 | 
					
 | 
				
			||||||
| 
						 | 
					@ -6,6 +6,7 @@ of the following people and organizations:
 | 
				
			||||||
 | 
					
 | 
				
			||||||
    Holger Krekel, holger at merlinux eu
 | 
					    Holger Krekel, holger at merlinux eu
 | 
				
			||||||
    Guido Wesdorp, johnny at johnnydebris net 
 | 
					    Guido Wesdorp, johnny at johnnydebris net 
 | 
				
			||||||
 | 
					    Samuele Pedroni, pedronis at openend se 
 | 
				
			||||||
    Carl Friedrich Bolz, cfbolz at gmx de
 | 
					    Carl Friedrich Bolz, cfbolz at gmx de
 | 
				
			||||||
    Armin Rigo, arigo at tunes org 
 | 
					    Armin Rigo, arigo at tunes org 
 | 
				
			||||||
    Maciek Fijalkowski, fijal at genesilico.pl
 | 
					    Maciek Fijalkowski, fijal at genesilico.pl
 | 
				
			||||||
| 
						 | 
					@ -14,7 +15,6 @@ of the following people and organizations:
 | 
				
			||||||
 | 
					
 | 
				
			||||||
Contributors include:: 
 | 
					Contributors include:: 
 | 
				
			||||||
 | 
					
 | 
				
			||||||
    Samuele Pedroni 
 | 
					 | 
				
			||||||
    Chris Lamb 
 | 
					    Chris Lamb 
 | 
				
			||||||
    Harald Armin Massa
 | 
					    Harald Armin Massa
 | 
				
			||||||
    Ralf Schmitt
 | 
					    Ralf Schmitt
 | 
				
			||||||
| 
						 | 
					
 | 
				
			||||||
		Loading…
	
		Reference in New Issue