Mailing List Archive

Greg> it recently occured to me that the 'spawn' module I wrote for the
Greg> Distutils (and which Perry Stoll extended to handle NT), could fit
Greg> nicely in the core library.

How's spawn.spawn semantically different from the Windows-dependent
os.spawn? How are stdout/stdin/stderr connected to the child process - just
like fork+exec or something slightly higher level like os.popen? If it's
semantically like os.spawn and a little bit higher level abstraction than
fork+exec, I'd vote for having the os module simply import it:

from spawn import spawn

and thus make that function more widely available...

Greg> The module as it's currently in the Distutils code is attached.

Not in the message I saw...

Skip Montanaro | http://www.mojam.com/
skip@mojam.com | http://www.musi-cal.com/~skip/
847-971-7098 | Python: Programming the way Guido indented...

On 30 August 1999, To python-dev@python.org said:
> The module as it's currently in the Distutils code is attached. Let me
> know what you think...

New definition of "attached": I'll just reply to my own message with the
code I meant to attach. D'oh!

------------------------------------------------------------------------
"""distutils.spawn

Provides the 'spawn()' function, a front-end to various platform-
specific functions for launching another program in a sub-process."""

# created 1999/07/24, Greg Ward

__rcsid__ = "$Id: spawn.py,v 1.2 1999/08/29 18:20:56 gward Exp $"

import sys, os, string
from distutils.errors import *


def spawn (cmd,
search_path=1,
verbose=0,
dry_run=0):

"""Run another program, specified as a command list 'cmd', in a new
process. 'cmd' is just the argument list for the new process, ie.
cmd[0] is the program to run and cmd[1:] are the rest of its
arguments. There is no way to run a program with a name different
from that of its executable.

If 'search_path' is true (the default), the system's executable
search path will be used to find the program; otherwise, cmd[0] must
be the exact path to the executable. If 'verbose' is true, a
one-line summary of the command will be printed before it is run.
If 'dry_run' is true, the command will not actually be run.

Raise DistutilsExecError if running the program fails in any way;
just return on success."""

if os.name == 'posix':
_spawn_posix (cmd, search_path, verbose, dry_run)
elif os.name in ( 'nt', 'windows' ): # ???
_spawn_nt (cmd, search_path, verbose, dry_run)
else:
raise DistutilsPlatformError, \
"don't know how to spawn programs on platform '%s'" % os.name

# spawn ()

def _spawn_nt ( cmd,
search_path=1,
verbose=0,
dry_run=0):
import string
executable = cmd[0]
if search_path:
paths = string.split( os.environ['PATH'], os.pathsep)
base,ext = os.path.splitext(executable)
if (ext != '.exe'):
executable = executable + '.exe'
if not os.path.isfile(executable):
paths.reverse() # go over the paths and keep the last one
for p in paths:
f = os.path.join( p, executable )
if os.path.isfile ( f ):
# the file exists, we have a shot at spawn working
executable = f
if verbose:
print string.join ( [executable] + cmd[1:], ' ')
if not dry_run:
# spawn for NT requires a full path to the .exe
rc = os.spawnv (os.P_WAIT, executable, cmd)
if rc != 0:
raise DistutilsExecError("command failed: %d" % rc)



def _spawn_posix (cmd,
search_path=1,
verbose=0,
dry_run=0):

if verbose:
print string.join (cmd, ' ')
if dry_run:
return
exec_fn = search_path and os.execvp or os.execv

pid = os.fork ()

if pid == 0: # in the child
try:
#print "cmd[0] =", cmd[0]
#print "cmd =", cmd
exec_fn (cmd[0], cmd)
except OSError, e:
sys.stderr.write ("unable to execute %s: %s\n" %
(cmd[0], e.strerror))
os._exit (1)

sys.stderr.write ("unable to execute %s for unknown reasons" % cmd[0])
os._exit (1)


else: # in the parent
# Loop until the child either exits or is terminated by a signal
# (ie. keep waiting if it's merely stopped)
while 1:
(pid, status) = os.waitpid (pid, 0)
if os.WIFSIGNALED (status):
raise DistutilsExecError, \
"command %s terminated by signal %d" % \
(cmd[0], os.WTERMSIG (status))

elif os.WIFEXITED (status):
exit_status = os.WEXITSTATUS (status)
if exit_status == 0:
return # hey, it succeeded!
else:
raise DistutilsExecError, \
"command %s failed with exit status %d" % \
(cmd[0], exit_status)

elif os.WIFSTOPPED (status):
continue

else:
raise DistutilsExecError, \
"unknown error executing %s: termination status %d" % \
(cmd[0], status)
# _spawn_posix ()
------------------------------------------------------------------------

--
Greg Ward - software developer gward@cnri.reston.va.us
Corporation for National Research Initiatives
1895 Preston White Drive voice: +1-703-620-8990
Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

On 30 August 1999, Skip Montanaro said:
>
> Greg> it recently occured to me that the 'spawn' module I wrote for the
> Greg> Distutils (and which Perry Stoll extended to handle NT), could fit
> Greg> nicely in the core library.
>
> How's spawn.spawn semantically different from the Windows-dependent
> os.spawn?

My understanding (purely from reading Perry's code!) is that the Windows
spawnv() and spawnve() calls require the full path of the executable,
and there is no spawnvp(). Hence, the bulk of Perry's '_spawn_nt()'
function is code to search the system path if the 'search_path' flag is
true.

In '_spawn_posix()', I just use either 'execv()' or 'execvp()'
for this. The bulk of my code is the complicated dance required to
wait for a fork'ed child process to finish.

> How are stdout/stdin/stderr connected to the child process - just
> like fork+exec or something slightly higher level like os.popen?

Just like fork 'n exec -- '_spawn_posix()' is just a front end to fork
and exec (either execv or execvp).

In a previous life, I *did* implement a spawning module for a certain
other popular scripting language that handles redirection and capturing
(backticks in the shell and that other scripting language). It was a
lot of fun, but pretty hairy. Took three attempts gradually developed
over two years to get it right in the end. In fact, it does all the
easy stuff that a Unix shell does in spawning commands, ie. search the
path, fork 'n exec, and redirection and capturing. Doesn't handle the
tricky stuff, ie. pipelines and job control.

The documentation for this module is 22 pages long; the code is 600+
lines of somewhat tricky Perl (1300 lines if you leave in comments and
blank lines). That's why the Distutils spawn module doesn't do anything
with std{out,err,in}.

> If it's semantically like os.spawn and a little bit higher level
> abstraction than fork+exec, I'd vote for having the os module simply
> import it:

So os.spawnv and os.spawnve would be Windows-specific, but os.spawn
portable? Could be confusing. And despite the recent extended
discussion of the os module, I'm not sure if this fits the model.

BTW, is there anything like this on the Mac? On what other OSs does it
even make sense to talk about programs spawning other programs? (Surely
those GUI user interfaces have to do *something*...)

Greg
--
Greg Ward - software developer gward@cnri.reston.va.us
Corporation for National Research Initiatives
1895 Preston White Drive voice: +1-703-620-8990
Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

Greg> BTW, is there anything like this on the Mac?

There will be, once Jack Jansen contributes _spawn_mac... ;-)

Skip Montanaro | http://www.mojam.com/
skip@mojam.com | http://www.musi-cal.com/~skip/
847-971-7098 | Python: Programming the way Guido indented...

Recently, Greg Ward <gward@cnri.reston.va.us> said:
> BTW, is there anything like this on the Mac? On what other OSs does it
> even make sense to talk about programs spawning other programs? (Surely
> those GUI user interfaces have to do *something*...)

Yes, but the interface is quite a bit more high-level, so it's pretty
difficult to reconcile with the Unix and Windows "every argument is a
string" paradigm. You start the process and pass along an AppleEvent
(basically an RPC-call) that will be presented to the program upon
startup.

So on the mac there's a serious difference between (inventing the API
interface here, cut down to make it understandable to non-macheads:-)
spawn("netscape", ("Open", "file.html"))
and
spawn("netscape", ("OpenURL", "http://foo.com/file.html"))

The mac interface is (of course:-) infinitely more powerful, allowing
you to talk to running apps, adressing stuff in it as COM/OLE does,
etc. but unfortunately the simple case of spawn("rm", "-rf", "/") is
impossible to represent in a meaningful way.

Add to that the fact that there's no stdin/stdout/stderr and there's
little common ground. The one area of common ground is "run program X
on files Y and Z and wait (or don't wait) for completion", so that is
something that could maybe have a special method that could be
implemented on all three mentioned platforms (and probably everything
else as well). And even then it'll be surprising to Mac users that
they have to _exit_ their editor (if you specify wait), not something
people commonly do.
--
Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

> Recently, Greg Ward <gward@cnri.reston.va.us> said:
> > BTW, is there anything like this on the Mac? On what other OSs does it
> > even make sense to talk about programs spawning other programs? (Surely
> > those GUI user interfaces have to do *something*...)
>
> Yes, but the interface is quite a bit more high-level, so it's pretty
> difficult to reconcile with the Unix and Windows "every argument is a
> string" paradigm. You start the process and pass along an AppleEvent
> (basically an RPC-call) that will be presented to the program upon
> startup.
>
> So on the mac there's a serious difference between (inventing the API
> interface here, cut down to make it understandable to non-macheads:-)
> spawn("netscape", ("Open", "file.html"))
> and
> spawn("netscape", ("OpenURL", "http://foo.com/file.html"))
>
> The mac interface is (of course:-) infinitely more powerful, allowing
> you to talk to running apps, adressing stuff in it as COM/OLE does,
> etc. but unfortunately the simple case of spawn("rm", "-rf", "/") is
> impossible to represent in a meaningful way.
>
> Add to that the fact that there's no stdin/stdout/stderr and there's
> little common ground. The one area of common ground is "run program X
> on files Y and Z and wait (or don't wait) for completion", so that is
> something that could maybe have a special method that could be
> implemented on all three mentioned platforms (and probably everything
> else as well). And even then it'll be surprising to Mac users that
> they have to _exit_ their editor (if you specify wait), not something
> people commonly do.

Indeed. I'm guessing that Greg wrote his code specifically to drive
compilers, not so much to invoke an editor on a specific file. It so
happens that the Windows compilers have command lines that look
sufficiently like the Unix compilers that this might actually work.

On the Mac, driving the compilers is best done using AppleEvents, so
it's probably better to to try to abuse the spawn() interface for
that... (Greg, is there a higher level where the compiler actions are
described without referring to specific programs, but perhaps just to
compiler actions and input and output files?)

--Guido van Rossum (home page: http://www.python.org/~guido/)

> it recently occured to me that the 'spawn' module I wrote for the
> Distutils (and which Perry Stoll extended to handle NT), could fit
> nicely in the core library. On Unix, it's just a front-end to
> fork-and-exec; on NT, it's a front-end to spawnv(). In either case,
> it's just enough code (and just tricky enough code) that not everybody
> should have to duplicate it for their own uses.
>
> The basic idea is this:
>
> from spawn import spawn
> ...
> spawn (['cmd', 'arg1', 'arg2'])
> # or
> spawn (['cmd'] + args)
>
> you get the idea: it takes a *list* representing the command to spawn:
> no strings to parse, no shells to get in the way, no sneaky
> meta-characters ruining your day, draining your efficiency, or
> compromising your security. (Conversely, no pipelines, redirection,
> etc.)
>
> The 'spawn()' function just calls '_spawn_posix()' or '_spawn_nt()'
> depending on os.name. Additionally, it takes a couple of optional
> keyword arguments (all booleans): 'search_path', 'verbose', and
> 'dry_run', which do pretty much what you'd expect.
>
> The module as it's currently in the Distutils code is attached. Let me
> know what you think...

I'm not sure that the verbose and dry_run options belong in the
standard library. When both are given, this does something
semi-useful; for Posix that's basically just printing the arguments,
while for NT it prints the exact command that will be executed. Not
sure if that's significant though.

Perhaps it's better to extract the code that runs the path to find the
right executable and make that into a separate routine. (Also, rather
than reversing the path, I would break out of the loop at the first
hit.)

--Guido van Rossum (home page: http://www.python.org/~guido/)

On 30 August 1999, Guido van Rossum said:
> Indeed. I'm guessing that Greg wrote his code specifically to drive
> compilers, not so much to invoke an editor on a specific file. It so
> happens that the Windows compilers have command lines that look
> sufficiently like the Unix compilers that this might actually work.

Correct, but the spawn module I posted should work for any case where
you want to run an external command synchronously without redirecting
I/O. (And it could probably be extended to handle those cases, but a) I
don't need them for Distutils [yet!], and b) I don't know how to do it
portably.)

> On the Mac, driving the compilers is best done using AppleEvents, so
> it's probably better to to try to abuse the spawn() interface for
> that... (Greg, is there a higher level where the compiler actions are
> described without referring to specific programs, but perhaps just to
> compiler actions and input and output files?)

[off-topic alert... probably belongs on distutils-sig, but there you go]
Yes, my CCompiler class is all about providing a (hopefully) compiler-
and platform-neutral interface to a C/C++ compiler. Currently there're
only two concrete subclasses of this: UnixCCompiler and MSVCCompiler,
and they both obviously use spawn, because Unix C compilers and MSVC
both provide that kind of interface. A hypothetical sibling class that
provides an interface to some Mac C compiler might use a souped-up spawn
that "knows about" Apple Events, or it might use some other interface to
Apple Events. If Jack's simplified summary of what passing Apple Events
to a command looks like is accurate, maybe spawn can be souped up to
work on the Mac. Or we might need a dedicated module for running Mac
programs.

So does anybody have code to run external programs on the Mac using
Apple Events? Would it be possible/reasonable to add that as
'_spawn_mac()' to my spawn module?

Greg
--
Greg Ward - software developer gward@cnri.reston.va.us
Corporation for National Research Initiatives
1895 Preston White Drive voice: +1-703-620-8990
Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

Hmm, if we're talking a "Python Make" or some such here the best way
would probably be to use Tool Server. Tool Server is a thing that is
based on Apple's old MPW programming environment, that is still
supported by compiler vendors like MetroWerks.

The nice thing of Tool Server for this type of work is that it _is_
command-line based, so you can probably send it things like
spawn("cc", "-O", "test.c")

But, although I know it is possible to do this with ToolServer, I
haven't a clue on how to do it...
--
Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

[Greg Ward]
> ...
> In a previous life, I *did* implement a spawning module for
> a certain other popular scripting language that handles
> redirection and capturing (backticks in the shell and that other
> scripting language). It was a lot of fun, but pretty hairy. Took
> three attempts gradually developed over two years to get it right
> in the end. In fact, it does all the easy stuff that a Unix shell
> does in spawning commands, ie. search the path, fork 'n exec, and
> redirection and capturing. Doesn't handle the tricky stuff, ie.
> pipelines and job control.
>
> The documentation for this module is 22 pages long; the code
> is 600+ lines of somewhat tricky Perl (1300 lines if you leave
> in comments and blank lines). That's why the Distutils spawn
> module doesn't do anything with std{out,err,in}.

Note that win/tclWinPipe.c-- which contains the Windows-specific support for
Tcl's "exec" cmd --is about 3,200 lines of C. It does handle pipelines and
redirection, and even fakes pipes as needed with temp files when it can
identify a pipeline component as belonging to the 16-bit subsystem. Even so,
the Tcl help page for "exec" bristles with hilarious caveats under the Windows
subsection; e.g.,

When redirecting from NUL:, some applications may hang, others
will get an infinite stream of "0x01" bytes, and some will
actually correctly get an immediate end-of-file; the behavior
seems to depend upon something compiled into the application
itself. When redirecting greater than 4K or so to NUL:, some
applications will hang. The above problems do not happen with
32-bit applications.

Still, people seem very happy with Tcl's exec, and I'm certain no language
tries harder to provide a portable way to "do command lines".

Two points to that:

1) If Python ever wants to do something similar, let's steal the Tcl code (&
unlike stealing Perl's code, stealing Tcl's code actually looks possible --
it's very much better organized and written).

2) For all its heroic efforts to hide platform limitations,

int
Tcl_ExecObjCmd(dummy, interp, objc, objv)
ClientData dummy; /* Not used. */
Tcl_Interp *interp; /* Current interpreter. */
int objc; /* Number of arguments. */
Tcl_Obj *CONST objv[]; /* Argument objects. */
{
#ifdef MAC_TCL

Tcl_AppendResult(interp, "exec not implemented under Mac OS",
(char *)NULL);
return TCL_ERROR;

#else
...

a-generalized-spawn-is-a-good-start-ly y'rs - tim

Greg Ward <gward@cnri.reston.va.us> wrote:
> it recently occured to me that the 'spawn' module I wrote for the
> Distutils (and which Perry Stoll extended to handle NT), could fit
> nicely in the core library. On Unix, it's just a front-end to
> fork-and-exec; on NT, it's a front-end to spawnv().

any reason this couldn't go into the os module instead?

just add parts of it to os.py, and change the docs to say
that spawn* are supported on Windows and Unix...

(supporting the full set of spawn* primitives would
of course be nice, btw. just like os.py provides all
exec variants...)

</F>