Mailing List Archive: forking + stdout = confusion

forking + stdout = confusion

clarence at silcom

Apr 12, 1999, 12:03 AM

Post #1 of 14 (2963 views)

I have a program running in Python 1.5.1 under FreeBSD 2.1.7.
It happens to be a nph-type CGI program. Normally, it runs a
report that can take quite a while, and I'm trying to add an
option to delay the start. I defined this function:

def LateShift():
OutputID = `os.getpid()`
Child = os.fork()
if Child:
print 'content-type: text/plain'
print
print '''The program is waiting to run later. The output will
be available with the id number of %s.
''' % OutputID
sys.exit(0)
sys.stdin.close()
sys.stdout.close()
sys.stderr.close()
sys.stdout = open(os.path.join(SpoolDir, OutputID), 'w')
sys.stderr = sys.stdout
time.sleep(15)

The idea is that, after forking, one of the processes will tell the
user how to get the report output later, then exit, and the other
will wait until some appropriate time and the run the report, with
the output going to a file that normally goes to the web browser.

The problem is that the browser's connection to the server is not
being closed until the actual report is done (i.e., until the caller
of this function exits). It seems to me that the three standard
I/O streams are closed (which they are immediately in both processes),
that should do it. Being an nph program, there should be no connection
to the HTTP server (which is Apache), but just in case it was waiting
for the child process to exit, I had the parent (Apache's child) exit
immediately and its child do the waiting, but it works the same either
way.

I thought that maybe sys.stdout was using a dup()'ed copy of stdout,
but I checked sys.stdout.fileno() and it was 1, so that doesn't seem
to be the case.

Can anyone hazard a guess (or see right off the top of his head) what's
going on here? Thanks for any help.

--
-=-=-=-=-=-=-=-=
Clarence Gardner
AvTel Communications
Software Products and Services Division
clarence@avtel.com

forking + stdout = confusion [ In reply to ]

Apr 12, 1999, 3:38 AM

Post #2 of 14 (2930 views)

Hi Clarence,

>>>>> "Clarence" == Clarence Gardner <clarence@silcom.com> writes:

[snip]

Clarence> def LateShift():
Clarence> OutputID = `os.getpid()`
Clarence> Child = os.fork()
Clarence> if Child:
Clarence> print 'content-type: text/plain'
Clarence> print
Clarence> print '''The program is waiting to run later. The output will
Clarence> be available with the id number of %s.
Clarence> ''' % OutputID
Clarence> sys.exit(0)
Clarence> sys.stdin.close()
Clarence> sys.stdout.close()
Clarence> sys.stderr.close()
Clarence> sys.stdout = open(os.path.join(SpoolDir, OutputID), 'w')
Clarence> sys.stderr = sys.stdout
Clarence> time.sleep(15)

[snip]

Hmm.. the following code works for me:

try:
serr = sys.stderr
sys.stderr = sys.stdout

sys.stdout.flush()
pid = os.fork()
if pid:
#
# OK...we are the parent process
#
sys.stdout.close()
sys.stdin.close()

os.wait()
else:
sys.stderr = serr
sys.stdout.close()
sys.stdin.close()

... on to do time consuming task.....

Clarence> I thought that maybe sys.stdout was using a dup()'ed
Clarence> copy of stdout, but I checked sys.stdout.fileno() and it
Clarence> was 1, so that doesn't seem to be the case.

Clarence> Can anyone hazard a guess (or see right off the top of
Clarence> his head) what's going on here? Thanks for any help.

I'm not sure, but I think sys.stdout is always 1. I think you need to close
sys.stdxxx in both processes explicitly. At least it works on my systems.

good luck!
-steve

Clarence> -- -=-=-=-=-=-=-=-= Clarence Gardner AvTel
Clarence> Communications Software Products and Services Division
Clarence> clarence@avtel.com

forking + stdout = confusion [ In reply to ]

fuzzy at fuzzys

Apr 12, 1999, 5:29 AM

Post #3 of 14 (2925 views)

>>>>> "Clarence" == Clarence Gardner <clarence@silcom.com> writes:

Clarence> I have a program running in Python 1.5.1 under FreeBSD 2.1.7.
Clarence> It happens to be a nph-type CGI program. Normally, it runs a
Clarence> report that can take quite a while, and I'm trying to add an
Clarence> option to delay the start. I defined this function:

[...]

I ran in exactly the same problem. I tried closing all stdxxx-filedescriptors,
setting a new session id with setsid, also tried it with setting a new
progress groups, and I even tried to reopen stdin, stderr and stdout to
/dev/null, but nothing helped. Then I saw that in OpenBSD, there's a function
in stdlib.h called "daemon()". You simply call it, and it puts the running
program in the background (it forks and exits the parent), redirects the
stdxxx-descriptors, sets a new session ID etc... and with this function, it
works!

Anyway, I was not able to reproduce what daemon does in any way, obviously it
does anything else which I did not think of, but I can't find out what. So for
now I use a very dirty wrapper which just calls the daemon-function, but since
I also realized that at least linux libc5 (I don't know how this is with
glibc) doesn't know this handy function, it would be fine to know what I
forgot while trying to reproduce daemon()'s actions.

I think it must be something with the stdxxx-filedescriptors and/or the
terminal, since using daemon() in a cgi lets apache close the connection to
the browser, but the simple call of fork, stdxxx.close/reopen/whatever, setsid
and setpgrp does not.

For know, check if you're system is having the daemon() function (it should
reside in the C-Library and its declaration in stdlib.h) and if it has, use it
either with a small C wrapper program, with a python module which implements
it or even using the dl-module.

--
/--/ Julien Oster /---/ www.fuzzys.org <---> www.sysadm.cc /---/
/--/ OpenBSD 2.5 /---/ Greetings from Munich, Germany /---/
/--/ contact me : /---/ talk fuzzy@fuzzys.org or e-Mail me /---/

forking + stdout = confusion [ In reply to ]

Apr 12, 1999, 10:38 AM

Post #4 of 14 (2923 views)

Julien Oster <fuzzy@fuzzys.org> writes:
...
| I ran in exactly the same problem. I tried closing all stdxxx-filedescriptors,
| setting a new session id with setsid, also tried it with setting a new
| progress groups, and I even tried to reopen stdin, stderr and stdout to
| /dev/null, but nothing helped. Then I saw that in OpenBSD, there's a function
| in stdlib.h called "daemon()". You simply call it, and it puts the running
| program in the background (it forks and exits the parent), redirects the
| stdxxx-descriptors, sets a new session ID etc... and with this function, it
| works!

I wonder if you could get the effect by iteratively closing all potential
file descriptors? This is a pretty common sight in service daemon programs.
In Python it's a little more awkward because you have to guess the range
of possible descriptors, which a C program could get from getdtablesize().

wereopen = []
for i in range(128):
try:
os.close(i)
wereopen.append(i)
except os.error:
pass
fp = open('log', 'w')
fp.write('closed units %s\n' % repr(wereopen))

The other things I've seen daemon() do are basically Berkeley terminal
driver issues, to drop the controlling terminal and isolate the program
from the job control session that started it -- not relevant there, since
the HTTP service daemon didn't start it that way.

Donn Cave, University Computing Services, University of Washington
donn@u.washington.edu

forking + stdout = confusion [ In reply to ]

Apr 13, 1999, 3:02 AM

Post #5 of 14 (2923 views)

clarence@silcom.com (Clarence Gardner) writes:

> I thought that maybe sys.stdout was using a dup()'ed copy of stdout,
> but I checked sys.stdout.fileno() and it was 1, so that doesn't seem
> to be the case.
No, the child and the parent share the SAME filedescriptors. This is true
for ALL open files! So if you don't need the descriptors in a child or a
parent you should close them.

Apache waits for the end of the file, so all processes (parents and
childs) must close the file!

greetings from germany
Carsten

forking + stdout = confusion [ In reply to ]

Apr 13, 1999, 8:55 AM

Post #6 of 14 (2930 views)

Carsten Schabacker <cs@spock.rhein-neckar.de> wrote:
> No, the child and the parent share the SAME filedescriptors. This is true
> for ALL open files! So if you don't need the descriptors in a child or a
> parent you should close them.
>
> Apache waits for the end of the file, so all processes (parents and
> childs) must close the file!

When doing a similar thing a while ago, I had similar problems :-) I
eventually found that in addition to doing sys.stdout.close(), I had to do
os.close(1) to make sure the fd really got closed.

I'm still not sure I understand what was going on. I know that
sys.__stdout__ contains a copy of sys.stdout, so at first I figured that
doing both sys.stdout.close() and sys.__stdout__.close() would do the
trick, but it didn't, but os.close(1) did.

--
Roy Smith <roy@popmail.med.nyu.edu>
New York University School of Medicine

forking + stdout = confusion [ In reply to ]

clarence at silcom

Apr 14, 1999, 8:11 AM

Post #7 of 14 (2946 views)

Roy Smith (roy@popmail.med.nyu.edu) wrote:
: Carsten Schabacker <cs@spock.rhein-neckar.de> wrote:
: > No, the child and the parent share the SAME filedescriptors. This is true
: > for ALL open files! So if you don't need the descriptors in a child or a
: > parent you should close them.
: >
: > Apache waits for the end of the file, so all processes (parents and
: > childs) must close the file!
Carsten: Please take this constructively :) I am aware of the fd sharing,
and the code I posted clearly closed stdin, stdout, and stderr in both the
parent and child processes. Also, in case you're not aware of it, the
distinguishing feature of a nph-type CGI program is that its output does
not go through the web server.

:
: When doing a similar thing a while ago, I had similar problems :-) I
: eventually found that in addition to doing sys.stdout.close(), I had to do
: os.close(1) to make sure the fd really got closed.
:
: I'm still not sure I understand what was going on. I know that
: sys.__stdout__ contains a copy of sys.stdout, so at first I figured that
: doing both sys.stdout.close() and sys.__stdout__.close() would do the
: trick, but it didn't, but os.close(1) did.

That does indeed do the trick, but you probably had to os.close(2) also.
(Both stdout and stderr normally are piped into Apache.)
However, your first thought also works, with the same caveat about stderr.
stdin, stdout, and stderr all have the __xxx__ copy in the sys module
(which I was not aware of).

So it was a Python issue after all! Removes any tinge of guilt I had
about posting it here :)

--
-=-=-=-=-=-=-=-=
Clarence Gardner
AvTel Communications
Software Products and Services Division
clarence@avtel.com

forking + stdout = confusion [ In reply to ]

Apr 14, 1999, 8:28 AM

Post #8 of 14 (2927 views)

clarence@silcom.com (Clarence Gardner) wrote:
> That does indeed do the trick, but you probably had to os.close(2) also.
> (Both stdout and stderr normally are piped into Apache.)

I don't know how all Apaches work, but at least on our machine, cgis get
pipes for stdin and stdout, but stderr is connected directly to the error
log file. I wrote a little test cgi which is just:

> #!/bin/sh
> sleep 300

opened http://my.host.name/cgi-bin/roy/sleeper.cgi and ran lsof, which is
a neat utility which mucks about in the kernel's file tables and shows you
what files a process has open. I got:

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sleep 25247 nobody 0r FIFO 0x04a6ff00 0 22787
sleep 25247 nobody 1w FIFO 0x04096e00 0 22788
sleep 25247 nobody 2w VREG 3049,395040 4420204 90
/www/logs/error_log

--
Roy Smith <roy@popmail.med.nyu.edu>
New York University School of Medicine

forking + stdout = confusion [ In reply to ]

Apr 14, 1999, 8:58 AM

Post #9 of 14 (2930 views)

clarence@silcom.com (Clarence Gardner) writes:
> Clarence Gardner (clarence@silcom.com) wrote:
> : However, your first thought also works, with the same caveat about stderr.
> : stdin, stdout, and stderr all have the __xxx__ copy in the sys module
> : (which I was not aware of).
>
> Mea culpa. The os.close() *is* still necessary. Is there yet another
> copy of these file objects? I tried to find that function that returns
> the reference count, but don't see it in the manual.

It's sys.refcount:

Python 1.5.2 (#2, Apr 14 1999, 13:02:03) [GCC egcs-2.91.66 19990314 (egcs-1.1.2 on linux2
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> import sys
>>> sys.getrefcount (sys.stdout )
5

Five! I don't know where all of those are...

> --
> -=-=-=-=-=-=-=-=
> Clarence Gardner
> AvTel Communications
> Software Products and Services Division
> clarence@avtel.com

forking + stdout = confusion [ In reply to ]

clarence at silcom

Apr 14, 1999, 9:36 AM

Post #10 of 14 (2925 views)

Clarence Gardner (clarence@silcom.com) wrote:
: However, your first thought also works, with the same caveat about stderr.
: stdin, stdout, and stderr all have the __xxx__ copy in the sys module
: (which I was not aware of).

Mea culpa. The os.close() *is* still necessary. Is there yet another
copy of these file objects? I tried to find that function that returns
the reference count, but don't see it in the manual.

--
-=-=-=-=-=-=-=-=
Clarence Gardner
AvTel Communications
Software Products and Services Division
clarence@avtel.com

forking + stdout = confusion [ In reply to ]

Apr 14, 1999, 10:04 AM

Post #11 of 14 (2923 views)

> >>> import sys
> >>> sys.getrefcount (sys.stdout )
> 5
>
> Five! I don't know where all of those are...

Yowza! That certainly explains a lot of the problems I had!

--
Roy Smith <roy@popmail.med.nyu.edu>
New York University School of Medicine

forking + stdout = confusion [ In reply to ]

Apr 15, 1999, 12:41 AM

Post #12 of 14 (2927 views)

clarence@silcom.com (Clarence Gardner) writes:

> Roy Smith (roy@popmail.med.nyu.edu) wrote:

> parent and child processes. Also, in case you're not aware of it, the
> distinguishing feature of a nph-type CGI program is that its output does
> not go through the web server.
Could you please explain me what a nph-type CGI is ? I know only
regular CGI-Scripts where stdout goes through the webserver.

Thanx Carsten.

forking + stdout = confusion [ In reply to ]

tim_one at email

Apr 15, 1999, 8:02 PM

Post #13 of 14 (2928 views)

[Clarence Gardner]
>> Mea culpa. The os.close() *is* still necessary. Is there yet another
>> copy of these file objects? I tried to find that function that returns
>> the reference count, but don't see it in the manual.

[Michael Hudson]
> It's sys.getrefcount:
>
> Python 1.5.2 (#2, Apr 14 1999, 13:02:03) [GCC egcs-2.91.66
> 19990314 (egcs-1.1.2 on linux2
> Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
> >>> import sys
> >>> sys.getrefcount (sys.stdout )
> 5
>
> Five! I don't know where all of those are...

Two are in sys (sys.stdout & sys.__stdout__), two more are in a hidden copy
of the initial contents of sys used to keep extension modules sane, and the
fifth is-- in effect --a temporary side-effect of passing the object to
sys.getrefcount itself (you may have noticed that sys.getrefcount never
returns a count less than 2!).

It doesn't matter, though, because unlike file objects you create with
"open", Python won't close the file descriptor underlying the default
binding for sys.stdout (or .stdin or .stderr) even if all references do go
away. Those three file objects are initialized specially (in _PySys_Init).

2-plus-2-equals-5-and-0-equals-3-ly y'rs - tim

forking + stdout = confusion [ In reply to ]

clarence at silcom

Apr 16, 1999, 8:04 AM

Post #14 of 14 (2925 views)

Tim Peters (tim_one@email.msn.com), as usual, knew the answer:
: It doesn't matter, though, because unlike file objects you create with
: "open", Python won't close the file descriptor underlying the default
: binding for sys.stdout (or .stdin or .stderr) even if all references do go
: away. Those three file objects are initialized specially (in _PySys_Init).

I promise this will be my last posting in this thread :)
Tim:
Thanks!

Carsten, et al.:
The Apache, CERN and Netscape HTTP servers (at least for Unix) treat
any CGI program differently if its name begins with 'nph-'. Normally,
stdout in a CGI is a pipe into the web server, which copies the data
to the client. Before the data, however, it also includes the HTTP
response headers (though it requires you to supply the Content-Type).
It may also buffer the entire response so that it can include a
Content-Length.

Technically, the difference in the NPH environment is that the server
does not add any headers, so the entire HTTP response must be provided
by the CGI program. (One of the reasons NPH is necessary is if you
want to provide an unusual HTTP response code. The server normally
takes care of Success and, if you provide a Location: or URI: header,
Redirection statuses.) (NPH supposedly means 'non parsed headers'.)

In practice, the server also does away with the buffering in the NPH
case, and sets the stdout for the CGI program to be the connection to
the client.

--
-=-=-=-=-=-=-=-=
Clarence Gardner
AvTel Communications
Software Products and Services Division
clarence@avtel.com