floraachy/pytest2
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[svn r37264] create the new development trunk

Browse Source 
--HG--
branch : trunk
This commit is contained in:
hpk
2007-01-24 15:24:01 +01:00
commit 5992a8ef21
435 changed files with 58640 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
py/doc/future/code_template.txt
+640
View File
@@ -0,0 +1,640 @@
===============================================================
py.code_template: Lightweight and flexible code template system
===============================================================
.. contents::
.. sectnum::
Motivation
==========
There are as many python templating systems as there are web frameworks
(a lot). This is partly because it is so darned easy to write a templating
system in Python. What are the distinguishing characteristics of the
py.code_template templating system?
* Optimized for generating code (Python, C, bash scripts, etc.),
not XML or HTML
* Designed for use by Python programmers, not by web artists
+ Aesthetic sensibilities are different
+ The templates should be an organic part of a module -- just more code
+ Templates do not need to be incredibly full-featured, because
programmers are perfectly capable of escaping to Python for
advanced features.
- No requirement to support inheritance
- No requirement to support exec
* Designed so that templates can be coded in the most natural way
for the task at hand
+ Generation of code and scripts often does not follow MVC paradigm!
+ Small template fragments are typically coded *inside* Python modules
+ Sometimes it is natural to put strings inside code; sometimes it is
natural to put code inside strings. Both should be supported as
reasonably and naturally as possible.
Imaginary-world examples
========================
These would be real-world examples, but, not only is this module not yet
implemented, as of now, PyPy is not incredibly useful to the average
programmer...
translator/c/genc.py
--------------------
The original function::
def gen_readable_parts_of_main_c_file(f, database, preimplementationlines=[]):
#
# All declarations
#
structdeflist = database.getstructdeflist()
print >> f
print >> f, '/***********************************************************/'
print >> f, '/*** Structure definitions ***/'
print >> f
for node in structdeflist:
print >> f, 'struct %s;' % node.name
print >> f
for node in structdeflist:
for line in node.definition():
print >> f, line
print >> f
print >> f, '/***********************************************************/'
print >> f, '/*** Forward declarations ***/'
print >> f
for node in database.globalcontainers():
for line in node.forward_declaration():
print >> f, line
#
# Implementation of functions and global structures and arrays
#
print >> f
print >> f, '/***********************************************************/'
print >> f, '/*** Implementations ***/'
print >> f
for line in preimplementationlines:
print >> f, line
print >> f, '#include "src/g_include.h"'
print >> f
blank = True
for node in database.globalcontainers():
if blank:
print >> f
blank = False
for line in node.implementation():
print >> f, line
blank = True
This could be refactored heavily. An initial starting point
would look something like this, although later, the template
instance could be passed in and reused directly, rather than
passing the file handle around::
def gen_readable_parts_of_main_c_file(f, database, preimplementationlines=[]):
def container_implementation():
# Helper function designed to introduce blank lines
# between container implementations
blank = True
for node in database.globalcontainers():
if blank:
yield ''
blank = False
for line in node.implementation():
yield line
blank = True
t = code_template.Template()
#
# All declarations
#
structdeflist = database.getstructdeflist()
t.write(dedent=8, text='''
/***********************************************************/
/*** Structure definitions ***/
{for node in structdeflist}
struct {node.name};
{endfor}
{for node in structdeflist}
{for line in node.definition}
{line}
{endfor}
{endfor}
/***********************************************************/
/*** Forward declarations ***/
{for node in database.globalcontainers()}
{for line in node.forward_declaration()}
{line}
{endfor}
{endfor}
{**
** Implementation of functions and global structures and arrays
**}
/***********************************************************/
/*** Implementations ***/
{for line in preimplementationlines}
{line}
{endfor}
#include "src/g_include.h"
{for line in container_implementation()}
{line}
{endfor}
""")
t.output(f)
translator/c/genc.py gen_makefile
---------------------------------
The original code::
MAKEFILE = '''
CC = gcc
$(TARGET): $(OBJECTS)
\t$(CC) $(LDFLAGS) -o $@ $(OBJECTS) $(LIBDIRS) $(LIBS)
%.o: %.c
\t$(CC) $(CFLAGS) -o $@ -c $< $(INCLUDEDIRS)
clean:
\trm -f $(OBJECTS)
'''
def gen_makefile(self, targetdir):
def write_list(lst, prefix):
for i, fn in enumerate(lst):
print >> f, prefix, fn,
if i < len(lst)-1:
print >> f, '\\'
else:
print >> f
prefix = ' ' * len(prefix)
compiler = self.getccompiler(extra_includes=['.'])
cfiles = []
ofiles = []
for fn in compiler.cfilenames:
fn = py.path.local(fn).basename
assert fn.endswith('.c')
cfiles.append(fn)
ofiles.append(fn[:-2] + '.o')
f = targetdir.join('Makefile').open('w')
print >> f, '# automatically generated Makefile'
print >> f
print >> f, 'TARGET =', py.path.local(compiler.outputfilename).basename
print >> f
write_list(cfiles, 'SOURCES =')
print >> f
write_list(ofiles, 'OBJECTS =')
print >> f
args = ['-l'+libname for libname in compiler.libraries]
print >> f, 'LIBS =', ' '.join(args)
args = ['-L'+path for path in compiler.library_dirs]
print >> f, 'LIBDIRS =', ' '.join(args)
args = ['-I'+path for path in compiler.include_dirs]
write_list(args, 'INCLUDEDIRS =')
print >> f
print >> f, 'CFLAGS =', ' '.join(compiler.compile_extra)
print >> f, 'LDFLAGS =', ' '.join(compiler.link_extra)
print >> f, MAKEFILE.strip()
f.close()
Could look something like this::
MAKEFILE = '''
# automatically generated Makefile
TARGET = {py.path.local(compiler.outputfilename).basename}
{for line in write_list(cfiles, 'SOURCES =')}
{line}
{endfor}
{for line in write_list(ofiles, 'OBJECTS =')}
{line}
{endfor}
LIBS ={for libname in compiler.libraries} -l{libname}{endfor}
LIBDIRS ={for path in compiler.library_dirs} -L{path}{endfor}
INCLUDEDIRS ={for path in compiler.include_dirs} -I{path}{endfor}
CFLAGS ={for extra in compiler.compile_extra} {extra}{endfor}
LDFLAGS ={for extra in compiler.link_extra} {extra}{endfor}
CC = gcc
$(TARGET): $(OBJECTS)
\t$(CC) $(LDFLAGS) -o $@ $(OBJECTS) $(LIBDIRS) $(LIBS)
%.o: %.c
\t$(CC) $(CFLAGS) -o $@ -c $< $(INCLUDEDIRS)
clean:
\trm -f $(OBJECTS)
'''
def gen_makefile(self, targetdir):
def write_list(lst, prefix):
for i, fn in enumerate(lst):
yield '%s %s %s' % (prefix, fn, i < len(list)-1 and '\\' or '')
prefix = ' ' * len(prefix)
compiler = self.getccompiler(extra_includes=['.'])
cfiles = []
ofiles = []
for fn in compiler.cfilenames:
fn = py.path.local(fn).basename
assert fn.endswith('.c')
cfiles.append(fn)
ofiles.append(fn[:-2] + '.o')
code_template.Template(MAKEFILE).output(targetdir.join('Makefile'))
translator/llvm/module/excsupport.py
------------------------------------
The original string::
invokeunwind_code = '''
ccc %(returntype)s%%__entrypoint__%(entrypointname)s {
%%result = invoke %(cconv)s %(returntype)s%%%(entrypointname)s to label %%no_exception except label %%exception
no_exception:
store %%RPYTHON_EXCEPTION_VTABLE* null, %%RPYTHON_EXCEPTION_VTABLE** %%last_exception_type
ret %(returntype)s %%result
exception:
ret %(noresult)s
}
ccc int %%__entrypoint__raised_LLVMException() {
%%tmp = load %%RPYTHON_EXCEPTION_VTABLE** %%last_exception_type
%%result = cast %%RPYTHON_EXCEPTION_VTABLE* %%tmp to int
ret int %%result
}
internal fastcc void %%unwind() {
unwind
}
'''
Could look something like this if it was used in conjunction with a template::
invokeunwind_code = '''
ccc {returntype}%__entrypoint__{entrypointname} {
%result = invoke {cconv} {returntype}%{entrypointname} to label %no_exception except label %exception
no_exception:
store %RPYTHON_EXCEPTION_VTABLE* null, %RPYTHON_EXCEPTION_VTABLE** %last_exception_type
ret {returntype} %result
exception:
ret {noresult}
}
ccc int %__entrypoint__raised_LLVMException() {
%tmp = load %RPYTHON_EXCEPTION_VTABLE** %last_exception_type
%result = cast %RPYTHON_EXCEPTION_VTABLE* %tmp to int
ret int %result
}
internal fastcc void %unwind() {
unwind
}
'''
Template syntax
===============
Design decision
---------------
As all programmers must know by now, all the special symbols on the keyboard
are quite heavily overloaded. Often, template systems work around this fact
by having special notation like `<*` ... `*>` or {% ... %}. Some template systems
even have multiple special notations -- one for comments, one for statements,
one for expressions, etc.
I find these hard to type and ugly. Other markups are either too lightweight,
or use characters which occur so frequently in the target languages that it
becomes hard to distinguish marked-up content from content which should be
rendered as-is.
The compromise taken by *code_template* is to use braces (**{}**) for markup.
This immediately raises the question: what about when the marked-up language
is C or C++? The answer is that if the leading brace is immediately followed
by whitespace, it is normal text; if not it is the start of markup.
To support normal text which has a leading brace immediately followed by
an identifier, if the first whitespace character after the brace is a space
character (e.g. not a newline or tab), it will be removed from the output.
Examples::
{ This is normal text and the space between { and This will be removed}
{'this must be a valid Python expression' + ' because it is treated as markup'}
{
This is normal text, but nothing is altered (the newline is kept intact)
}
{{1:'Any valid Python expression is allowed as markup'}[1].ljust(30)}
.. _`Code element`:
Elements
--------
Templates consist of normal text and code elements.
(Comments are considered to be code elements.)
All code elements start with a `left brace`_ which is not followed by
whitespace.
Keyword element
~~~~~~~~~~~~~~~
A keyword element is a `code element`_ which starts with a keyword_.
For example, *{if foo}* is a keyword element, but *{foo}* is a `substituted expression`_.
Keyword
~~~~~~~
A keyword is a word used in `conditional text`_ or in `repeated text`_, e.g.
one of *if*, *elif*, *else*, *endif*, *for*, or *endfor*.
Keywords are designed to match their Python equivalents. However, since
templates cannot use spacing to indicate expression nesting, the additional
keywords *endif* and *endfor* are required.
Left brace
~~~~~~~~~~
All elements other than normal text start with a left brace -- the symbol '{',
sometimes known as a 'curly bracket'. A left brace is itself considered
to be normal text if it is followed by whitespace. If the whitespace starts
with a space character, that space character will be stripped from the output.
If the whitespace starts with a tab or linefeed character, the whitespace will
be left in the output.
Normal Text
~~~~~~~~~~~
Normal text remains unsubstituted. Transition from text to the other elements
is effected by use of a `left brace`_ which is not followed by whitespace.
Comment
~~~~~~~
A comment starts with a left brace followed by an asterisk ('{`*`'), and
ends with an asterisk followed by a right brace ('`*`}')::
This is a template -- this text will be copied to the output.
{* This is a comment and this text will not be copied to the output *}
{*
Comments can span lines,
but cannot be nested
*}
Substituted expression
~~~~~~~~~~~~~~~~~~~~~~
Any python expression may be used::
Dear {record.name},
we are sorry to inform you that you did not win {record.contest}.
The expression must be surrounded by braces, and there must not be any
whitespace between the leftmost brace and the start of the expression.
The expression will automatically be converted to a string with str().
Conditional text
~~~~~~~~~~~~~~~~
The following template has text which is included conditionally::
This text will always be included in the output
{if foo}
This text will be included if foo is true
{elif bar}
This text will be included if foo is not true but bar is true
{else}
This text will be included if neither foo nor bar is true
{endif}
The {elif} and {else} elements are optional.
Repeated text
~~~~~~~~~~~~~
The following template shows how to pull multiple items out of a list::
{for student, score in sorted(scorelist)}
{student.ljust(20)} {score}
{endfor}
Whitespace removal or modification
----------------------------------
In general, whitespace in `Normal Text`_ is transferred unchanged to the
output. There are three exceptions to this rule:
Line separators
~~~~~~~~~~~~~~~
Each newline is converted to the final output using os.linesep.
Beginning or end of string
~~~~~~~~~~~~~~~~~~~~~~~~~~
py.code_template is designed to allow easy use of templates inside of python
modules. The canonical way to write a template is inside a triple-quoted
string, e.g.::
my_template = '''
This is my template. It can have any text at all in it except
another triple-single-quote.
'''
To support this usage, if the first character is a newline, it will be
removed, and if the last line consists solely of whitespace with no
trailing newline, it will also be removed.
A comment or single keyword element on a line
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Whenever a `keyword element`_ or comment_ is on a line
*by itself*, that line will not be copied to the output.
This happens when:
- There is nothing on the line before the keyword element
or comment except whitespace (spaces and/or tabs).
- There is nothing on the line after the keyword element
or comment except a newline.
Note that even a multi-line comment or keyword element can
have the preceding whitespace and subsequent newline stripped
by this rule.
The primary purpose of this rule is to allow the Python
programmer to use indentation, **even inside a template**::
This is a template
{if mylist}
List items:
{for item in mylist}
- {item}
{endfor}
{endif}
Template usage
==============
Templates are used by importing the Template class from py.code_template,
constructing a template, and then sending data with the write() method.
In general, there are four methods for getting the formatted data back out
of the template object:
- read() reads all the data currently in the object
- output(fobj) outputs the data to a file
fobj can either be an open file object, or a string. If it is
a string, the file will be opened, written, and closed.
- open(fobj) (or calling the object constructor with a file object)
If the open() method is used, or if a file object is passed to
the constructor, each write() will automatically flush the data
out to the file. If the fobj is a string, it is considered to
be *owned*, otherwise it is considered to be *borrowed*. *Owned*
file objects are closed when the class is deleted.
- write() can be explicitly called with a file object, in which case
it will invoke output() on that object after it generates the data.
Template instantiation and methods
==================================
template = code_template.Template(outf=None, cache=None)
If outf is given, it will be passed to the open() method
cache may be given as a mapping. If not given, the template will use
the shared default cache. This is not thread safe.
template.open
-------------
template.open(outf, borrowed = None)
The open method closes the internal file object if it was already open,
and then re-opens it on the given file. It is an error to call open()
if there is data in the object left over from previous writes. (Call
output() instead.)
borrowed defaults to 0 if outf is a string, and 1 if it is a file object.
borrowed can also be set explicitly if required.
template.close
--------------
close() disassociates the file from the template, and closes the file if
it was not borrowed. close() is automatically called by the destructor.
template.write
--------------
template.write(text='', outf=None, dedent=0, localvars=None, globalvars=None,
framelevel=1)
The write method has the following parameters:
- text is the template itself
- if outf is not None, the output method will be invoked on the object
after the current template is processed. If no outf is given, data
will be accumulated internal to the instance until a write() with outf
is processed, or read() or output() is called, whichever comes first, if
there is no file object. If there is a file object, data will be flushed
to the file after every write.
- dedent, if given is applied to each line in the template, to "de-indent"
- localvars and globalvars default to the dictionaries of the caller. A copy
of localvars is made so that the __TrueSpace__ identifier can be added.
- cache may be given as a mapping. If not given, the template will use
the shared default cache. This is not thread safe.
- framelevel is used to determine which stackframe to access for globals
and locals if localvars and/or globalvars are not specified. The default
is to use the caller's frame.
The write method supports the print >> file protocol by deleting the softspace
attribute on every invocation. This allows code like::
t = code_template.Template()
print >> t, "Hello, world"
template.read
--------------
This method reads and flushes all accumulated data in the object. Note that
if a file has been associated with the object, there will never be any data
to read.
template.output
---------------
This method takes one parameter, outf. template.output() first
invokes template.read() to read and flush all accumulated data,
and then outputs the data to the file specified by outf.
If outf has a write() method, that will be invoked with the
data. If outf has no write() method, it will be treated as
a filename, and that file will be replaced.
Caching and thread safety
=========================
The compiled version of every template is cached internal to the
code_template module (unless a separate cache object is specified).
This allows efficient template reuse, but is not currently thread-safe.
Alternatively, each invocation of a template object can specify a
cache object. This is thread-safe, but not very efficient. A shared
model could be implemented later.
py/doc/future/future.txt
+423
View File
@@ -0,0 +1,423 @@
=======================================================
Visions and ideas for further development of the py lib
=======================================================
.. contents::
.. sectnum::
This document tries to describe directions and guiding ideas
for the near-future development of the py lib. *Note that all
statements within this document - even if they sound factual -
mostly just express thoughts and ideas. They not always refer to
real code so read with some caution. This is not a reference guide
(tm). Moreover, the order in which appear here in the file does
not reflect the order in which they may be implemented.*
.. _`general-path`:
.. _`a more general view on path objects`:
A more general view on ``py.path`` objects
==========================================
Seen from a more general persective, the current ``py.path.extpy`` path
offers a way to go from a file to the structured content of
a file, namely a python object. The ``extpy`` path retains some
common ``path`` operations and semantics but offers additional
methods, e.g. ``resolve()`` gets you a true python object.
But apart from python files there are many other examples
of structured content like xml documents or INI-style
config files. While some tasks will only be convenient
to perform in a domain specific manner (e.g. applying xslt
etc.pp) ``py.path`` offers a common behaviour for
structured content paths. So far only ``py.path.extpy``
is implemented and used by py.test to address tests
and traverse into test files.
*You are in a maze of twisty passages, all alike*
-------------------------------------------------
Now, for the sake of finding out a good direction,
let's consider some code that wants to find all
*sections* which have a certain *option* value
within some given ``startpath``::
def find_option(startpath, optionname):
for section in startpath.listdir(dir=1):
opt = section.join(optionname)
if opt.check(): # does the option exist here?
print section.basename, "found:", opt.read()
Now the point is that ``find_option()`` would obviously work
when ``startpath`` is a filesystem-like path like a local
filesystem path or a subversion URL path. It would then see
directories as sections and files as option-names and the
content of the file as values.
But it also works (today) for ``extpy`` paths if you put the following
python code in a file::
class Section1:
someoption = "i am an option value"
class Section2:
someoption = "i am another option value"
An ``extpy()`` path maps classes and modules to directories and
name-value bindings to file/read() operations.
And it could also work for 'xml' paths if you put
the following xml string in a file::
<xml ...>
<root>
<section1>
<someoption>value</name></section1>
<section2>
<someoption>value</name></section2></root>
where tags containing non-text tags map to directories
and tags with just text-children map to files (which
upon read() return the joined content of the text
tags possibly as unicode.
Now, to complete the picture, we could make Config-Parser
*ini-style* config files also available::
[section1]
name = value
[section2]
othername = value
where sections map to directories and name=value mappings
to file/contents.
So it seems that our above ``find_option()`` function would
work nicely on all these *mappings*.
Of course, the somewhat open question is how to make the
transition from a filesystem path to structured content
useful and unified, as much as possible without overdoing it.
Again, there are tasks that will need fully domain specific
solutions (DOM/XSLT/...) but i think the above view warrants
some experiments and refactoring. The degree of uniformity
still needs to be determined and thought about.
path objects should be stackable
--------------------------------
Oh, and btw, a ``py.path.extpy`` file could live on top of a
'py.path.xml' path as well, i.e. take::
<xml ...>
<code>
<py>
<magic>
<assertion>
import py
... </assertion>
<exprinfo>
def getmsg(x): pass </exprino></magic></py></code>
and use it to have a ``extpy`` path living on it::
p = py.path.local(xmlfilename)
xmlp = py.path.extxml(p, 'py/magic/exprinfo')
p = py.path.extpy(xmlp, 'getmsg')
assert p.check(func=1, basename='getmsg')
getmsg = p.resolve()
# we now have a *live* getmsg() function taken and compiled from
# the above xml fragment
There could be generic converters which convert between
different content formats ... allowing configuration files to e.g.
be in XML/Ini/python or filesystem-format with some common way
to find and iterate values.
*After all the unix filesystem and the python namespaces are
two honking great ideas, why not do more of them? :-)*
.. _importexport:
Revising and improving the import/export system
===============================================
or let's wrap the world all around
the export/import interface
---------------------------
The py lib already incorporates a mechanism to select which
namespaces and names get exposed to a user of the library.
Apart from reducing the outside visible namespaces complexity
this allows to quickly rename and refactor stuff in the
implementation without affecting the caller side. This export
control can be used by other python packages as well.
However, all is not fine as the import/export has a
few major deficiencies and shortcomings:
- it doesn't allow to specify doc-strings
- it is a bit hackish (see py/initpkg.py)
- it doesn't present a complete and consistent view of the API.
- ``help(constructed_namespace)`` doesn't work for the root
package namespace
- when the py lib implementation accesses parts of itself
it uses the native python import mechanism which is
limiting in some respects. Especially for distributed
programs as encouraged by `py.execnet`_ it is not clear
how the mechanism can nicely integrate to support remote
lazy importing.
Discussions have been going on for a while but it is
still not clear how to best tackle the problem. Personally,
i believe the main missing thing for the first major release
is the docstring one. The current specification
of exported names is dictionary based. It would be
better to declare it in terms of Objects.
Example sketch for a new export specification
---------------------------------------------
Here is a sketch of how the py libs ``__init__.py`` file
might or should look like::
"""
the py lib version 0.8
http://codespeak.net/py/0.8
"""
from py import pkg
pkg.export(__name__,
pkg.Module('path',
'''provides path objects for local filesystem,
subversion url and working copy, and extension paths.
''',
pkg.Class('local', '''
the local filesystem path offering a single
point of interaction for many purposes.
''', extpy='./path/local.LocalPath'),
pkg.Class('svnurl', '''
the subversion url path.
''', extpy='./path/local/svn/urlcommand.SvnUrlPath'),
),
# it goes on ...
)
The current ``initpkg.py`` code can be cleaned up to support
this new more explicit style of stating things. Note that
in principle there is nothing that stops us from retrieving
implementations over the network, e.g. a subversion repository.
Let there be alternatives
-------------------------
We could also specify alternative implementations easily::
pkg.Class('svnwc', '''
the subversion working copy.
''', extpy=('./path/local/svn/urlbinding.SvnUrlPath',
'./path/local/svn/urlcommand.SvnUrlPath',)
)
This would prefer the python binding based implementation over
the one working through he 'svn' command line utility. And
of course, it could uniformly signal if no implementation is
available at all.
Problems problems
-----------------
Now there are reasons there isn't a clear conclusion so far.
For example, the above approach has some implications, the
main one being that implementation classes like
``py/path/local.LocalPath`` are visible to the caller side but
this presents an inconsistency because the user started out with
``py.path.local`` and expects that the two classes are really much
the same. We have the same problem today, of course.
The naive solution strategy of wrapping the "implementation
level" objects into their exported representations may remind
of the `wrapping techniques PyPy uses`_. But it
*may* result in a slightly heavyweight mechanism that affects
runtime speed. However, I guess that this standard strategy
is probably the cleanest.
Every problem can be solved with another level ...
--------------------------------------------------
The wrapping of implementation level classes in their export
representations objects adds another level of indirection.
But this indirection would have interesting advantages:
- we could easily present a consistent view of the library
- it could take care of exceptions as well
- it provides natural interception points for logging
- it enables remote lazy loading of implementations
or certain versions of interfaces
And quite likely the extra indirection wouldn't hurt so much
as it is not much more than a function call and we cared
we could even generate some c-code (with PyPy :-) to speed
it up.
But it can lead to new problems ...
-----------------------------------
However, it is critical to avoid to burden the implementation
code of being aware of its wrapping. This is what we have
to do in PyPy but the import/export mechanism works at
a higher level of the language, i think.
Oh, and we didn't talk about bootstrapping :-)
.. _`py.execnet`: ../execnet.html
.. _`wrapping techniques PyPy uses`: http://codespeak.net/pypy/index.cgi?doc/wrapping.html
.. _`lightweight xml generation`:
Extension of py.path.local.sysexec()
====================================
The `sysexec mechanism`_ allows to directly execute
binaries on your system. Especially after we'll have this
nicely integrated into Win32 we may also want to run python
scripts both locally and from the net::
vadm = py.path.svnurl('http://codespeak.net/svn/vadm/dist/vadm/cmdline.py')
stdoutput = vadm.execute('diff')
To be able to execute this code fragement, we need either or all of
- an improved import system that allows remote imports
- a way to specify what the "neccessary" python import
directories are. for example, the above scriptlet will
require a certain root included in the python search for module
in order to execute something like "import vadm".
- a way to specify dependencies ... which opens up another
interesting can of worms, suitable for another chapter
in the neverending `future book`_.
.. _`sysexec mechanism`: ../misc.html#sysexec
.. _`compile-on-the-fly`:
we need a persistent storage for the py lib
-------------------------------------------
A somewhat open question is where to store the underlying
generated pyc-files and other files generated on the fly
with `CPython's distutils`_. We want to have a
*persistent location* in order to avoid runtime-penalties
when switching python versions and platforms (think NFS).
A *persistent location* for the py lib would be a good idea
maybe also for other reasons. We could cache some of the
expensive test setups, like the multi-revision subversion
repository that is created for each run of the tests.
.. _`CPython's distutils`: http://www.python.org/dev/doc/devel/lib/module-distutils.html
.. _`getting started`: ../getting-started.html
.. _`restructured text`: http://docutils.sourceforge.net/docs/user/rst/quickref.html
.. _`python standard library`: http://www.python.org/doc/2.3.4/lib/lib.html
.. _`xpython EuroPython 2004 talk`: http://codespeak.net/svn/user/hpk/talks/xpython-talk.txt
.. _`under the xpy tree`: http://codespeak.net/svn/user/hpk/xpy/xml.py
.. _`future book`: future.html
.. _`PEP-324 subprocess module`: http://www.python.org/peps/pep-0324.html
.. _`subprocess implementation`: http://www.lysator.liu.se/~astrand/popen5/
.. _`py.test`: ../test.html
Refactor path implementations to use a Filesystem Abstraction
=============================================================
It seems like a good idea to refactor all python implementations to
use an internal Filesystem abstraction. The current code base
would be transformed to have Filesystem implementations for e.g.
local, subversion and subversion "working copy" filesystems. Today
the according code is scattered through path-handling code.
On a related note, Armin Rigo has hacked `pylufs`_ which allows to
implement kernel-level linux filesystems with pure python. Now
the idea is that the mentioned filesystem implementations would
be directly usable for such linux-filesystem glue code.
In other words, implementing a `memoryfs`_ or a `dictfs`_ would
give you two things for free: a filesystem mountable at kernel level
as well as a uniform "path" object allowing you to access your
filesystem in convenient ways. (At some point it might
even become interesting to think about interfacing to
`reiserfs v4 features`_ at the Filesystem level but that
is a can of subsequent worms).
.. _`memoryfs`: http://codespeak.net/svn/user/arigo/hack/pyfuse/memoryfs.py
.. _`dictfs`: http://codespeak.net/pipermail/py-dev/2005-January/000191.html
.. _`pylufs`: http://codespeak.net/svn/user/arigo/hack/pylufs/
.. _`reiserfs v4 features`: http://www.namesys.com/v4/v4.html
Improve and unify Path API
==========================
visit() grows depth control
---------------------------
Add a ``maxdepth`` argument to the path.visit() method,
which will limit traversal to subdirectories. Example::
x = py.path.local.get_tmproot()
for x in p.visit('bin', stop=N):
...
This will yield all file or directory paths whose basename
is 'bin', depending on the values of ``stop``::
p # stop == 0 or higher (and p.basename == 'bin')
p / bin # stop == 1 or higher
p / ... / bin # stop == 2 or higher
p / ... / ... / bin # stop == 3 or higher
The default for stop would be `255`.
But what if `stop < 0`? We could let that mean to go upwards::
for x in x.visit('py/bin', stop=-255):
# will yield all parent direcotires which have a
# py/bin subpath
visit() returning a lazy list?
------------------------------
There is a very nice "no-API" `lazy list`_ implementation from
Armin Rigo which presents a complete list interface, given some
iterable. The iterable is consumed only on demand and retains
memory efficiency as much as possible. The lazy list
provides a number of advantages in addition to the fact that
a list interface is nicer to deal with than an iterator.
For example it lets you do::
for x in p1.visit('*.cfg') + p2.visit('*.cfg'):
# will iterate through all results
Here the for-iter expression will retain all lazyness (with
the result of adding lazy lists being another another lazy
list) by internally concatenating the underlying
lazylists/iterators. Moreover, the lazylist implementation
will know that there are no references left to the lazy list
and throw away iterated elements. This makes the iteration
over the sum of the two visit()s as efficient as if we had
used iterables to begin with!
For this, we would like to move the lazy list into the
py lib's namespace, most probably at `py.builtin.lazylist`.
.. _`lazy list`: http://codespeak.net/svn/user/arigo/hack/misc/collect.py
py/doc/future/planning.txt
+66
View File
@@ -0,0 +1,66 @@
Release
=======
currently working configurations
--------------------------------
2.3 - 2.4.2 work
2.5 has obscure problems
with setuptools: 2.3 - 2.4.2 as 'develop'
regular installation: works mostly, strange test-failures
to be tested: 2.2, windows
absolutely necessary steps:
----------------------------
* documentation
* improving getting started, describe install methods
* describe the rest stuff?
* py.log
* py.path is mostly undocumented, API documentation
* basic windows testing, maybe disabling execnet?, what about the scripts in windows?
* are all c extensions compiled when installing globally?
* refactoring py.log
* write/read methods on py.path should be renamed/deprecated: setcontent, getcontent instead?
* what about _subprocess.c?
* warning for docutils
* don't expose _extpy
* py/bin should be nicefied, get optparse interface
* _findpy.py
* py.cleanup:
* py.lookup: add -i option
* pytest.cmd
* rst2pdf.py: merge with py.rest, add warnings when missing tex
* _makepyrelease.py: move somewhere
* py.countloc
* py.test
* py.rest
* win32
* skip tests if dependencies are not installed
nice to have
------------
* sets.py, subprocess.py in compat
* fix -k option to py.test
* add --report=(text|terminal|session|rest|tkinter|rest) to py.test
* put Armin's collect class into py.__builtin__ (not done)
* try get rid of Collect.tryiter() in favour of (not done)
using Armin's collect class
py/doc/future/pylib_pypy.txt
+37
View File
@@ -0,0 +1,37 @@
Here I'm trying to list all problems regarding pypy-c <-> pylib interaction
===========================================================================
* in test/terminal/terminal.py lines around 141::
rev = py.__package__.getrev()
self.out.line("using py lib: %s <rev %s>" % (
py.path.local(py.__file__).dirpath(), rev))
* py.code issues::
def __init__(self, rawcode):
rawcode = getattr(rawcode, 'im_func', rawcode)
rawcode = getattr(rawcode, 'func_code', rawcode)
self.raw = rawcode
self.filename = rawcode.co_filename
AttributeError: 'internal-code' object has no attribute 'co_filename'
* types.BuiltinFunctionType == types.MethodType which confuses apigen
* compiler module problems - some bogus IndentationError
communicates by inspect.getsource()
* execnet just hangs
* lack of tmpfile
* assertion error magic is not working
* sha counting hangs (misc/testing/test\_initpkg)
* extpy does not work, because it does not support loops in modules
(while pypy __builtins__ module has a loop), funny :-)
* py.compat.subprocess hangs for obscure reasons
(possibly the same stuff as execnet - some threading issues and
select.select)
Armin says: "haha, select.select probably does not release the GIL"
py/doc/future/rsession_todo.txt
+15
View File
@@ -0,0 +1,15 @@
Various tasks which needs to be done at some point
==================================================
* Write down pinging interface, so we'll know if hosts are responding or
are mostly down (detecting hanging nodes)
* Write down support for rsync progress
* Discovery of nodes which are available for accepting distributed testing
* Test the tests rescheduling, so high-latency nodes would not take part
in that.
* make sure that C-c semantics are ok (nodes are killed properly).
There was an attempt to do so, but it's not tested and not always work.