[svn r63212] completely revamp the execnet web page

--HG--
branch : trunk

py/doc/execnet.txt

@@ -1,57 +1,27 @@
-The py.execnet library 
+
-======================
+
+``py.execnet`` allows to:
+
+* instantiate local or remote Python Processes
+* send code for execution in one or many processes 
+* asynchronously send and receive data between processes through channels 
+* completely avoid manual installation steps on remote places
 
 .. contents::
-.. sectnum::
 
-A new view on distributed execution
+Gateways: immediately spawn local or remote process
------------------------------------ 
+----------------------------------------------------
 
-``py.execnet`` supports ad-hoc distribution of parts of
+In order to send code to a remote place or a subprocess
-a program across process and network barriers.  *Ad-hoc*
+you need to instantiate a so-called Gateway object.  
-means that the client side may completely control 
+There are currently three Gateway classes:
 
-* which parts of a program execute remotely and 
+* :api:`py.execnet.PopenGateway` to open a subprocess 
+  on the local machine.  Useful for making use 
+  of multiple processors to to contain code execution
+  in a separated environment. 
 
-* which data protocols are used between them 
+* :api:`py.execnet.SshGateway` to connect to 
-
-without requiring any prior manual installation 
-of user program code on the remote side.  In fact,
-not even a prior installation of any server code
-is required, provided there is a way to get 
-an input/output connection to a python interpreter
-(for example via "ssh" and a "python" executable). 
-
-By comparison, traditional Remote Method Based (RMI) 
-require prior installation and manual rather
-heavy processes of setup, distribution and 
-communication between program parts.  
-
-
-What about Security? Are you completely nuts? 
----------------------------------------------
-
-We'll talk about that later :-) 
-
-Basic Features
-==============
-
-With ''py.execnet'' you get the means 
-
-- to execute python code fragements in remote processes and 
-- to interchange data between asynchronously executing code fragments 
-
-
-Available Gateways
------------------------------------------
-
-You may use one of the following connection methods:
-
-* :api:`py.execnet.PopenGateway` a subprocess on the local 
-  machine.  Useful for jailing certain parts of a program
-  or for making use of multiple processors. 
-
-* :api:`py.execnet.SshGateway` a way to connect to 
   a remote ssh server and distribute execution to it. 
 
 * :api:`py.execnet.SocketGateway` a way to connect to 
@@ -59,13 +29,14 @@ You may use one of the following connection methods:
   requires a manually started 
   :source:py/execnet/script/socketserver.py
   script.  You can run this "server script" without 
-  having the py lib installed on that remote system. 
+  having the py lib installed on the remote system
+  and you can setup it up as permanent service. 
 
 
-executing code remotely 
+remote_exec: execute source code remotely 
--------------------------------------
+--------------------------------------------
 
-All gateways offer remote code execution via this high level function:
+All gateways offer remote code execution via this high level function::
 
     def remote_exec(source): 
         """return channel object for communicating with the asynchronously 
@@ -76,7 +47,7 @@ With `remote_exec` you send source code to the other
 side and get both a local and a remote Channel_ object,
 which you can use to have the local and remote site
 communicate data in a structured way.   Here is 
-an example: 
+an example for reading the PID::
 
   >>> import py 
   >>> gw = py.execnet.PopenGateway()
@@ -92,7 +63,7 @@ an example:
 .. _`channel-api`: 
 .. _`exchange data`: 
 
-Bidirectionally exchange data between hosts 
+Channels: bidirectionally exchange data between hosts 
 -------------------------------------------------------------
 
 A channel object allows to send and receive data between 
@@ -131,44 +102,52 @@ Here is the interface of channel objects::
         A remote side blocking on receive() on this channel 
         will get woken up and see an EOFError exception. 
 
-Instantiating a gateway from a string-specification
----------------------------------------------------------
 
-To specify Gateways with a String::
+.. _xspec:
-
-    >>> import py
-    >>> gw = py.execnet.makegateway("popen")
-    >>> ex = gw.remote_exec("import sys ; channel.send(sys.executable)").receive()
-    >>> assert ex == py.std.sys.executable, (ex, py.std.sys.executable)
-    >>>
-
-current gateway types and specifications
-+++++++++++++++++++++++++++++++++++++++++++++++
-
-+------------------------+-------------------------------------------+
-| ssh host               | ssh:host:pythonexecutable:path            |
-+------------------------+-------------------------------------------+
-| local subprocess       | popen:python_executable:path              |
-+------------------------+-------------------------------------------+
-| remote socket process  | socket:host:port:python_executable:path   |
-+------------------------+-------------------------------------------+
-
-examples of valid specifications
-++++++++++++++++++++++++++++++++++++++
-
-``ssh:wyvern:python2.4`` signifies a connection to a Python process on the machine reached via "ssh wyvern", current dir will be the 'pyexecnet-cache' subdirectory. 
-
-``socket:192.168.1.4`` signifies a connection to a Python Socket server process to the given IP on the default port 8888; current dir will be the 'pyexecnet-cache' directory. 
-
-``popen:python2.5`` signifies a connection to a python2.5 subprocess; current dir will be the current dir of the instantiator. 
-
-``popen::pytest-cache1`` signifies a connection to a subprocess using ``sys.executable``; current dir will be the `pytest-cache1`. 
 
 
-Examples for execnet usage
+XSpec: string specification for gateway type and configuration
--------------------------------------------
+-------------------------------------------------------------------
 
-Example: compare cwd() of Popen Gateways
+``py.execnet`` supports a simple extensible format for 
+specifying and configuring Gateways for remote execution.  
+You can use a string speficiation to make a new gateway, 
+for example a new SshGateway::
+
+    gateway = py.execnet.makegateway("ssh=myhost")
+
+Let's look at some examples for valid specifications. 
+Specification for an ssh connection to `wyvern`, running on python2.4 in the (newly created) 'mycache'  subdirectory::
+
+    ssh=wyvern//python=python2.4//chdir=mycache
+
+Specification of a python2.5 subprocess; with a low CPU priority ("nice" level). Current dir will be the current dir of the instantiator (that's true for all 'popen' specifications unless they specify 'chdir')::
+
+    popen//python=2.5//nice=20
+
+Specification of a Python Socket server process that listens on 192.168.1.4:8888; current dir will be the 'pyexecnet-cache' sub directory which is used a default for all remote processes::
+
+    socket=192.168.1.4:8888
+
+More generally, a specification string has this general format::
+
+    key1=value1//key2=value2//key3=value3
+
+If you omit a value, a boolean true value is assumed.  Currently
+the following key/values are supported: 
+
+* ``popen`` for a PopenGateway
+* ``ssh=host`` for a SshGateway
+* ``socket=address:port`` for a SocketGateway 
+* ``python=executable`` for specifying Python executables
+* ``chdir=path`` change remote working dir to given relative or absolute path
+* ``nice=value`` decrease remote nice level if platforms supports it 
+  
+
+Examples fo py.execnet usage 
+-------------------------------------
+
+compare cwd() of Popen Gateways
 ++++++++++++++++++++++++++++++++++++++++
 
 A PopenGateway has the same working directory as the instantiatior::
@@ -180,10 +159,10 @@ A PopenGateway has the same working directory as the instantiatior::
     >>> assert res == os.getcwd()
     >>> gw.exit()
 
-Example: multichannels 
+synchronously receive results from two sub processes 
-++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
-MultiChannels manage 1-n execution and communication:
+Use MultiChannels for receiving multiple results from remote code::
 
     >>> import py
     >>> ch1 = py.execnet.PopenGateway().remote_exec("channel.send(1)")
@@ -193,20 +172,33 @@ MultiChannels manage 1-n execution and communication:
     >>> assert len(l) == 2
     >>> assert 1 in l 
     >>> assert 2 in l 
+   
+assynchronously receive results from two sub processes 
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
-MultiGateways help with sending code to multiple remote places: 
+Use ``MultiChannel.make_receive_queue()`` for asynchronously receiving 
+multiple results from remote code.  This standard Queue provides 
+``(channel, result)`` tuples which allows to determine where 
+a result comes from::
 
     >>> import py
-    >>> mgw = py.execnet.MultiGateway([py.execnet.PopenGateway() for x in range(2)])
+    >>> ch1 = py.execnet.PopenGateway().remote_exec("channel.send(1)")
-    >>> mch = mgw.remote_exec("import os; channel.send(os.getcwd())")
+    >>> ch2 = py.execnet.PopenGateway().remote_exec("channel.send(2)")
-    >>> res = mch.receive_each()
+    >>> mch = py.execnet.MultiChannel([ch1, ch2])
-    >>> assert res == [os.getcwd()] * 2, res
+    >>> queue = mch.make_receive_queue()
-    >>> mgw.exit()
+    >>> chan1, res1 = queue.get()  # you may also specify a timeout 
+    >>> chan2, res2 = queue.get()
+    >>> res1 + res2 
+    3
+    >>> assert chan1 in (ch1, ch2)
+    >>> assert chan2 in (ch1, ch2)
+    >>> assert chan1 != chan2
 
-
+receive file contents from remote SSH account 
-Example: receiving file contents from remote places
 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-problem: retrieving contents of remote files::
+
+Here is a small program that you can use to retrieve
+contents of remote files::
 
     import py
     # open a gateway to a fresh child process 
@@ -224,50 +216,20 @@ problem: retrieving contents of remote files::
         # process content 
      
     # later you can exit / close down the gateway
-    contentgateway.exit()
+    gw.exit()
 
 
-A more complicated "nested" Gateway Example 
+Instantiate a socket server in a new subprocess 
-........................................... 
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
 The following example opens a PopenGateway, i.e. a python
-child process, starts a socket server within that process and
+child process, and starts a socket server within that process 
-then opens a SocketGateway to the freshly started
+and then opens a second gateway to the freshly started
-socketserver.  Thus it forms a "triangle":: 
+socketserver::
-
+                
-
-    CLIENT < ... >  PopenGateway() 
-        <             .
-         .            .
-          .          .
-           .        .
-            > SocketGateway() 
-            
-The below "socketserver" mentioned script is a small script that 
-basically listens and accepts socket connections, receives one 
-liners and executes them. 
-
-Here are 20 lines of code making the above triangle happen::
-
     import py 
-    port = 7770
-    socketserverbootstrap = py.code.Source(
-        mypath.dirpath().dirpath('bin', 'socketserver.py').read(),
-        """
-        import socket
-        sock = bind_and_listen(("localhost", %r)) 
-        channel.send("ok") 
-        startserver(sock)
-    """ % port) 
-    # open a gateway to a fresh child process
-    proxygw = py.execnet.PopenGateway()
-
-    # execute asynchronously the above socketserverbootstrap on the other
-    channel = proxygw.remote_exec(socketserverbootstrap)
-
-    # the other side should start the socket server now 
-    assert channel.receive() == "ok" 
-    gw = py.execnet.SocketGateway('localhost', cls.port)
-    print "initialized socket gateway to port", cls.port
 
+    popengw = py.execnet.PopenGateway()
+    socketgw = py.execnet.SocketGateway.new_remote(popengw, ("127.0.0.1", 0))
 
+    print socketgw._rinfo() # print some info about the remote environment