diff --git a/py/doc/execnet.txt b/py/doc/execnet.txt index 9d297d370..1e6fe30e8 100644 --- a/py/doc/execnet.txt +++ b/py/doc/execnet.txt @@ -1,57 +1,27 @@ -The py.execnet library -====================== + + +``py.execnet`` allows to: + +* instantiate local or remote Python Processes +* send code for execution in one or many processes +* asynchronously send and receive data between processes through channels +* completely avoid manual installation steps on remote places .. contents:: -.. sectnum:: -A new view on distributed execution ------------------------------------ +Gateways: immediately spawn local or remote process +---------------------------------------------------- -``py.execnet`` supports ad-hoc distribution of parts of -a program across process and network barriers. *Ad-hoc* -means that the client side may completely control +In order to send code to a remote place or a subprocess +you need to instantiate a so-called Gateway object. +There are currently three Gateway classes: -* which parts of a program execute remotely and +* :api:`py.execnet.PopenGateway` to open a subprocess + on the local machine. Useful for making use + of multiple processors to to contain code execution + in a separated environment. -* which data protocols are used between them - -without requiring any prior manual installation -of user program code on the remote side. In fact, -not even a prior installation of any server code -is required, provided there is a way to get -an input/output connection to a python interpreter -(for example via "ssh" and a "python" executable). - -By comparison, traditional Remote Method Based (RMI) -require prior installation and manual rather -heavy processes of setup, distribution and -communication between program parts. - - -What about Security? Are you completely nuts? ---------------------------------------------- - -We'll talk about that later :-) - -Basic Features -============== - -With ''py.execnet'' you get the means - -- to execute python code fragements in remote processes and -- to interchange data between asynchronously executing code fragments - - -Available Gateways ------------------------------------------ - -You may use one of the following connection methods: - -* :api:`py.execnet.PopenGateway` a subprocess on the local - machine. Useful for jailing certain parts of a program - or for making use of multiple processors. - -* :api:`py.execnet.SshGateway` a way to connect to +* :api:`py.execnet.SshGateway` to connect to a remote ssh server and distribute execution to it. * :api:`py.execnet.SocketGateway` a way to connect to @@ -59,13 +29,14 @@ You may use one of the following connection methods: requires a manually started :source:py/execnet/script/socketserver.py script. You can run this "server script" without - having the py lib installed on that remote system. + having the py lib installed on the remote system + and you can setup it up as permanent service. -executing code remotely -------------------------------------- +remote_exec: execute source code remotely +-------------------------------------------- -All gateways offer remote code execution via this high level function: +All gateways offer remote code execution via this high level function:: def remote_exec(source): """return channel object for communicating with the asynchronously @@ -76,7 +47,7 @@ With `remote_exec` you send source code to the other side and get both a local and a remote Channel_ object, which you can use to have the local and remote site communicate data in a structured way. Here is -an example: +an example for reading the PID:: >>> import py >>> gw = py.execnet.PopenGateway() @@ -92,7 +63,7 @@ an example: .. _`channel-api`: .. _`exchange data`: -Bidirectionally exchange data between hosts +Channels: bidirectionally exchange data between hosts ------------------------------------------------------------- A channel object allows to send and receive data between @@ -131,44 +102,52 @@ Here is the interface of channel objects:: A remote side blocking on receive() on this channel will get woken up and see an EOFError exception. -Instantiating a gateway from a string-specification ---------------------------------------------------------- -To specify Gateways with a String:: - - >>> import py - >>> gw = py.execnet.makegateway("popen") - >>> ex = gw.remote_exec("import sys ; channel.send(sys.executable)").receive() - >>> assert ex == py.std.sys.executable, (ex, py.std.sys.executable) - >>> - -current gateway types and specifications -+++++++++++++++++++++++++++++++++++++++++++++++ - -+------------------------+-------------------------------------------+ -| ssh host | ssh:host:pythonexecutable:path | -+------------------------+-------------------------------------------+ -| local subprocess | popen:python_executable:path | -+------------------------+-------------------------------------------+ -| remote socket process | socket:host:port:python_executable:path | -+------------------------+-------------------------------------------+ - -examples of valid specifications -++++++++++++++++++++++++++++++++++++++ - -``ssh:wyvern:python2.4`` signifies a connection to a Python process on the machine reached via "ssh wyvern", current dir will be the 'pyexecnet-cache' subdirectory. - -``socket:192.168.1.4`` signifies a connection to a Python Socket server process to the given IP on the default port 8888; current dir will be the 'pyexecnet-cache' directory. - -``popen:python2.5`` signifies a connection to a python2.5 subprocess; current dir will be the current dir of the instantiator. - -``popen::pytest-cache1`` signifies a connection to a subprocess using ``sys.executable``; current dir will be the `pytest-cache1`. +.. _xspec: -Examples for execnet usage -------------------------------------------- +XSpec: string specification for gateway type and configuration +------------------------------------------------------------------- -Example: compare cwd() of Popen Gateways +``py.execnet`` supports a simple extensible format for +specifying and configuring Gateways for remote execution. +You can use a string speficiation to make a new gateway, +for example a new SshGateway:: + + gateway = py.execnet.makegateway("ssh=myhost") + +Let's look at some examples for valid specifications. +Specification for an ssh connection to `wyvern`, running on python2.4 in the (newly created) 'mycache' subdirectory:: + + ssh=wyvern//python=python2.4//chdir=mycache + +Specification of a python2.5 subprocess; with a low CPU priority ("nice" level). Current dir will be the current dir of the instantiator (that's true for all 'popen' specifications unless they specify 'chdir'):: + + popen//python=2.5//nice=20 + +Specification of a Python Socket server process that listens on 192.168.1.4:8888; current dir will be the 'pyexecnet-cache' sub directory which is used a default for all remote processes:: + + socket=192.168.1.4:8888 + +More generally, a specification string has this general format:: + + key1=value1//key2=value2//key3=value3 + +If you omit a value, a boolean true value is assumed. Currently +the following key/values are supported: + +* ``popen`` for a PopenGateway +* ``ssh=host`` for a SshGateway +* ``socket=address:port`` for a SocketGateway +* ``python=executable`` for specifying Python executables +* ``chdir=path`` change remote working dir to given relative or absolute path +* ``nice=value`` decrease remote nice level if platforms supports it + + +Examples fo py.execnet usage +------------------------------------- + +compare cwd() of Popen Gateways ++++++++++++++++++++++++++++++++++++++++ A PopenGateway has the same working directory as the instantiatior:: @@ -180,10 +159,10 @@ A PopenGateway has the same working directory as the instantiatior:: >>> assert res == os.getcwd() >>> gw.exit() -Example: multichannels -++++++++++++++++++++++++++++++++++++++++ +synchronously receive results from two sub processes +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -MultiChannels manage 1-n execution and communication: +Use MultiChannels for receiving multiple results from remote code:: >>> import py >>> ch1 = py.execnet.PopenGateway().remote_exec("channel.send(1)") @@ -193,20 +172,33 @@ MultiChannels manage 1-n execution and communication: >>> assert len(l) == 2 >>> assert 1 in l >>> assert 2 in l + +assynchronously receive results from two sub processes ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -MultiGateways help with sending code to multiple remote places: +Use ``MultiChannel.make_receive_queue()`` for asynchronously receiving +multiple results from remote code. This standard Queue provides +``(channel, result)`` tuples which allows to determine where +a result comes from:: >>> import py - >>> mgw = py.execnet.MultiGateway([py.execnet.PopenGateway() for x in range(2)]) - >>> mch = mgw.remote_exec("import os; channel.send(os.getcwd())") - >>> res = mch.receive_each() - >>> assert res == [os.getcwd()] * 2, res - >>> mgw.exit() + >>> ch1 = py.execnet.PopenGateway().remote_exec("channel.send(1)") + >>> ch2 = py.execnet.PopenGateway().remote_exec("channel.send(2)") + >>> mch = py.execnet.MultiChannel([ch1, ch2]) + >>> queue = mch.make_receive_queue() + >>> chan1, res1 = queue.get() # you may also specify a timeout + >>> chan2, res2 = queue.get() + >>> res1 + res2 + 3 + >>> assert chan1 in (ch1, ch2) + >>> assert chan2 in (ch1, ch2) + >>> assert chan1 != chan2 - -Example: receiving file contents from remote places +receive file contents from remote SSH account ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -problem: retrieving contents of remote files:: + +Here is a small program that you can use to retrieve +contents of remote files:: import py # open a gateway to a fresh child process @@ -224,50 +216,20 @@ problem: retrieving contents of remote files:: # process content # later you can exit / close down the gateway - contentgateway.exit() + gw.exit() -A more complicated "nested" Gateway Example -........................................... +Instantiate a socket server in a new subprocess +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ The following example opens a PopenGateway, i.e. a python -child process, starts a socket server within that process and -then opens a SocketGateway to the freshly started -socketserver. Thus it forms a "triangle":: - - - CLIENT < ... > PopenGateway() - < . - . . - . . - . . - > SocketGateway() - -The below "socketserver" mentioned script is a small script that -basically listens and accepts socket connections, receives one -liners and executes them. - -Here are 20 lines of code making the above triangle happen:: - +child process, and starts a socket server within that process +and then opens a second gateway to the freshly started +socketserver:: + import py - port = 7770 - socketserverbootstrap = py.code.Source( - mypath.dirpath().dirpath('bin', 'socketserver.py').read(), - """ - import socket - sock = bind_and_listen(("localhost", %r)) - channel.send("ok") - startserver(sock) - """ % port) - # open a gateway to a fresh child process - proxygw = py.execnet.PopenGateway() - - # execute asynchronously the above socketserverbootstrap on the other - channel = proxygw.remote_exec(socketserverbootstrap) - - # the other side should start the socket server now - assert channel.receive() == "ok" - gw = py.execnet.SocketGateway('localhost', cls.port) - print "initialized socket gateway to port", cls.port + popengw = py.execnet.PopenGateway() + socketgw = py.execnet.SocketGateway.new_remote(popengw, ("127.0.0.1", 0)) + print socketgw._rinfo() # print some info about the remote environment