[svn r63212] completely revamp the execnet web page
--HG-- branch : trunk
This commit is contained in:
hpk
parent
2c2bfb5513
commit
5da714131c
|
|
@ -1,57 +1,27 @@
|
|||
The py.execnet library
|
||||
======================
|
||||
|
||||
|
||||
``py.execnet`` allows to:
|
||||
|
||||
* instantiate local or remote Python Processes
|
||||
* send code for execution in one or many processes
|
||||
* asynchronously send and receive data between processes through channels
|
||||
* completely avoid manual installation steps on remote places
|
||||
|
||||
.. contents::
|
||||
.. sectnum::
|
||||
|
||||
A new view on distributed execution
|
||||
-----------------------------------
|
||||
Gateways: immediately spawn local or remote process
|
||||
----------------------------------------------------
|
||||
|
||||
``py.execnet`` supports ad-hoc distribution of parts of
|
||||
a program across process and network barriers. *Ad-hoc*
|
||||
means that the client side may completely control
|
||||
In order to send code to a remote place or a subprocess
|
||||
you need to instantiate a so-called Gateway object.
|
||||
There are currently three Gateway classes:
|
||||
|
||||
* which parts of a program execute remotely and
|
||||
* :api:`py.execnet.PopenGateway` to open a subprocess
|
||||
on the local machine. Useful for making use
|
||||
of multiple processors to to contain code execution
|
||||
in a separated environment.
|
||||
|
||||
* which data protocols are used between them
|
||||
|
||||
without requiring any prior manual installation
|
||||
of user program code on the remote side. In fact,
|
||||
not even a prior installation of any server code
|
||||
is required, provided there is a way to get
|
||||
an input/output connection to a python interpreter
|
||||
(for example via "ssh" and a "python" executable).
|
||||
|
||||
By comparison, traditional Remote Method Based (RMI)
|
||||
require prior installation and manual rather
|
||||
heavy processes of setup, distribution and
|
||||
communication between program parts.
|
||||
|
||||
|
||||
What about Security? Are you completely nuts?
|
||||
---------------------------------------------
|
||||
|
||||
We'll talk about that later :-)
|
||||
|
||||
Basic Features
|
||||
==============
|
||||
|
||||
With ''py.execnet'' you get the means
|
||||
|
||||
- to execute python code fragements in remote processes and
|
||||
- to interchange data between asynchronously executing code fragments
|
||||
|
||||
|
||||
Available Gateways
|
||||
-----------------------------------------
|
||||
|
||||
You may use one of the following connection methods:
|
||||
|
||||
* :api:`py.execnet.PopenGateway` a subprocess on the local
|
||||
machine. Useful for jailing certain parts of a program
|
||||
or for making use of multiple processors.
|
||||
|
||||
* :api:`py.execnet.SshGateway` a way to connect to
|
||||
* :api:`py.execnet.SshGateway` to connect to
|
||||
a remote ssh server and distribute execution to it.
|
||||
|
||||
* :api:`py.execnet.SocketGateway` a way to connect to
|
||||
|
|
@ -59,13 +29,14 @@ You may use one of the following connection methods:
|
|||
requires a manually started
|
||||
:source:py/execnet/script/socketserver.py
|
||||
script. You can run this "server script" without
|
||||
having the py lib installed on that remote system.
|
||||
having the py lib installed on the remote system
|
||||
and you can setup it up as permanent service.
|
||||
|
||||
|
||||
executing code remotely
|
||||
-------------------------------------
|
||||
remote_exec: execute source code remotely
|
||||
--------------------------------------------
|
||||
|
||||
All gateways offer remote code execution via this high level function:
|
||||
All gateways offer remote code execution via this high level function::
|
||||
|
||||
def remote_exec(source):
|
||||
"""return channel object for communicating with the asynchronously
|
||||
|
|
@ -76,7 +47,7 @@ With `remote_exec` you send source code to the other
|
|||
side and get both a local and a remote Channel_ object,
|
||||
which you can use to have the local and remote site
|
||||
communicate data in a structured way. Here is
|
||||
an example:
|
||||
an example for reading the PID::
|
||||
|
||||
>>> import py
|
||||
>>> gw = py.execnet.PopenGateway()
|
||||
|
|
@ -92,7 +63,7 @@ an example:
|
|||
.. _`channel-api`:
|
||||
.. _`exchange data`:
|
||||
|
||||
Bidirectionally exchange data between hosts
|
||||
Channels: bidirectionally exchange data between hosts
|
||||
-------------------------------------------------------------
|
||||
|
||||
A channel object allows to send and receive data between
|
||||
|
|
@ -131,44 +102,52 @@ Here is the interface of channel objects::
|
|||
A remote side blocking on receive() on this channel
|
||||
will get woken up and see an EOFError exception.
|
||||
|
||||
Instantiating a gateway from a string-specification
|
||||
---------------------------------------------------------
|
||||
|
||||
To specify Gateways with a String::
|
||||
|
||||
>>> import py
|
||||
>>> gw = py.execnet.makegateway("popen")
|
||||
>>> ex = gw.remote_exec("import sys ; channel.send(sys.executable)").receive()
|
||||
>>> assert ex == py.std.sys.executable, (ex, py.std.sys.executable)
|
||||
>>>
|
||||
|
||||
current gateway types and specifications
|
||||
+++++++++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
+------------------------+-------------------------------------------+
|
||||
| ssh host | ssh:host:pythonexecutable:path |
|
||||
+------------------------+-------------------------------------------+
|
||||
| local subprocess | popen:python_executable:path |
|
||||
+------------------------+-------------------------------------------+
|
||||
| remote socket process | socket:host:port:python_executable:path |
|
||||
+------------------------+-------------------------------------------+
|
||||
|
||||
examples of valid specifications
|
||||
++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
``ssh:wyvern:python2.4`` signifies a connection to a Python process on the machine reached via "ssh wyvern", current dir will be the 'pyexecnet-cache' subdirectory.
|
||||
|
||||
``socket:192.168.1.4`` signifies a connection to a Python Socket server process to the given IP on the default port 8888; current dir will be the 'pyexecnet-cache' directory.
|
||||
|
||||
``popen:python2.5`` signifies a connection to a python2.5 subprocess; current dir will be the current dir of the instantiator.
|
||||
|
||||
``popen::pytest-cache1`` signifies a connection to a subprocess using ``sys.executable``; current dir will be the `pytest-cache1`.
|
||||
.. _xspec:
|
||||
|
||||
|
||||
Examples for execnet usage
|
||||
-------------------------------------------
|
||||
XSpec: string specification for gateway type and configuration
|
||||
-------------------------------------------------------------------
|
||||
|
||||
Example: compare cwd() of Popen Gateways
|
||||
``py.execnet`` supports a simple extensible format for
|
||||
specifying and configuring Gateways for remote execution.
|
||||
You can use a string speficiation to make a new gateway,
|
||||
for example a new SshGateway::
|
||||
|
||||
gateway = py.execnet.makegateway("ssh=myhost")
|
||||
|
||||
Let's look at some examples for valid specifications.
|
||||
Specification for an ssh connection to `wyvern`, running on python2.4 in the (newly created) 'mycache' subdirectory::
|
||||
|
||||
ssh=wyvern//python=python2.4//chdir=mycache
|
||||
|
||||
Specification of a python2.5 subprocess; with a low CPU priority ("nice" level). Current dir will be the current dir of the instantiator (that's true for all 'popen' specifications unless they specify 'chdir')::
|
||||
|
||||
popen//python=2.5//nice=20
|
||||
|
||||
Specification of a Python Socket server process that listens on 192.168.1.4:8888; current dir will be the 'pyexecnet-cache' sub directory which is used a default for all remote processes::
|
||||
|
||||
socket=192.168.1.4:8888
|
||||
|
||||
More generally, a specification string has this general format::
|
||||
|
||||
key1=value1//key2=value2//key3=value3
|
||||
|
||||
If you omit a value, a boolean true value is assumed. Currently
|
||||
the following key/values are supported:
|
||||
|
||||
* ``popen`` for a PopenGateway
|
||||
* ``ssh=host`` for a SshGateway
|
||||
* ``socket=address:port`` for a SocketGateway
|
||||
* ``python=executable`` for specifying Python executables
|
||||
* ``chdir=path`` change remote working dir to given relative or absolute path
|
||||
* ``nice=value`` decrease remote nice level if platforms supports it
|
||||
|
||||
|
||||
Examples fo py.execnet usage
|
||||
-------------------------------------
|
||||
|
||||
compare cwd() of Popen Gateways
|
||||
++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
A PopenGateway has the same working directory as the instantiatior::
|
||||
|
|
@ -180,10 +159,10 @@ A PopenGateway has the same working directory as the instantiatior::
|
|||
>>> assert res == os.getcwd()
|
||||
>>> gw.exit()
|
||||
|
||||
Example: multichannels
|
||||
++++++++++++++++++++++++++++++++++++++++
|
||||
synchronously receive results from two sub processes
|
||||
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
MultiChannels manage 1-n execution and communication:
|
||||
Use MultiChannels for receiving multiple results from remote code::
|
||||
|
||||
>>> import py
|
||||
>>> ch1 = py.execnet.PopenGateway().remote_exec("channel.send(1)")
|
||||
|
|
@ -193,20 +172,33 @@ MultiChannels manage 1-n execution and communication:
|
|||
>>> assert len(l) == 2
|
||||
>>> assert 1 in l
|
||||
>>> assert 2 in l
|
||||
|
||||
assynchronously receive results from two sub processes
|
||||
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
MultiGateways help with sending code to multiple remote places:
|
||||
Use ``MultiChannel.make_receive_queue()`` for asynchronously receiving
|
||||
multiple results from remote code. This standard Queue provides
|
||||
``(channel, result)`` tuples which allows to determine where
|
||||
a result comes from::
|
||||
|
||||
>>> import py
|
||||
>>> mgw = py.execnet.MultiGateway([py.execnet.PopenGateway() for x in range(2)])
|
||||
>>> mch = mgw.remote_exec("import os; channel.send(os.getcwd())")
|
||||
>>> res = mch.receive_each()
|
||||
>>> assert res == [os.getcwd()] * 2, res
|
||||
>>> mgw.exit()
|
||||
>>> ch1 = py.execnet.PopenGateway().remote_exec("channel.send(1)")
|
||||
>>> ch2 = py.execnet.PopenGateway().remote_exec("channel.send(2)")
|
||||
>>> mch = py.execnet.MultiChannel([ch1, ch2])
|
||||
>>> queue = mch.make_receive_queue()
|
||||
>>> chan1, res1 = queue.get() # you may also specify a timeout
|
||||
>>> chan2, res2 = queue.get()
|
||||
>>> res1 + res2
|
||||
3
|
||||
>>> assert chan1 in (ch1, ch2)
|
||||
>>> assert chan2 in (ch1, ch2)
|
||||
>>> assert chan1 != chan2
|
||||
|
||||
|
||||
Example: receiving file contents from remote places
|
||||
receive file contents from remote SSH account
|
||||
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||
problem: retrieving contents of remote files::
|
||||
|
||||
Here is a small program that you can use to retrieve
|
||||
contents of remote files::
|
||||
|
||||
import py
|
||||
# open a gateway to a fresh child process
|
||||
|
|
@ -224,50 +216,20 @@ problem: retrieving contents of remote files::
|
|||
# process content
|
||||
|
||||
# later you can exit / close down the gateway
|
||||
contentgateway.exit()
|
||||
gw.exit()
|
||||
|
||||
|
||||
A more complicated "nested" Gateway Example
|
||||
...........................................
|
||||
Instantiate a socket server in a new subprocess
|
||||
++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
The following example opens a PopenGateway, i.e. a python
|
||||
child process, starts a socket server within that process and
|
||||
then opens a SocketGateway to the freshly started
|
||||
socketserver. Thus it forms a "triangle"::
|
||||
|
||||
|
||||
CLIENT < ... > PopenGateway()
|
||||
< .
|
||||
. .
|
||||
. .
|
||||
. .
|
||||
> SocketGateway()
|
||||
|
||||
The below "socketserver" mentioned script is a small script that
|
||||
basically listens and accepts socket connections, receives one
|
||||
liners and executes them.
|
||||
|
||||
Here are 20 lines of code making the above triangle happen::
|
||||
|
||||
child process, and starts a socket server within that process
|
||||
and then opens a second gateway to the freshly started
|
||||
socketserver::
|
||||
|
||||
import py
|
||||
port = 7770
|
||||
socketserverbootstrap = py.code.Source(
|
||||
mypath.dirpath().dirpath('bin', 'socketserver.py').read(),
|
||||
"""
|
||||
import socket
|
||||
sock = bind_and_listen(("localhost", %r))
|
||||
channel.send("ok")
|
||||
startserver(sock)
|
||||
""" % port)
|
||||
# open a gateway to a fresh child process
|
||||
proxygw = py.execnet.PopenGateway()
|
||||
|
||||
# execute asynchronously the above socketserverbootstrap on the other
|
||||
channel = proxygw.remote_exec(socketserverbootstrap)
|
||||
|
||||
# the other side should start the socket server now
|
||||
assert channel.receive() == "ok"
|
||||
gw = py.execnet.SocketGateway('localhost', cls.port)
|
||||
print "initialized socket gateway to port", cls.port
|
||||
|
||||
popengw = py.execnet.PopenGateway()
|
||||
socketgw = py.execnet.SocketGateway.new_remote(popengw, ("127.0.0.1", 0))
|
||||
|
||||
print socketgw._rinfo() # print some info about the remote environment
|
||||
|
|
|
|||
Loading…
Reference in New Issue