Mailing List Archive: Question about envvars

Question about envvars

Jun 7, 1999, 7:34 AM

Post #1 of 7 (1109 views)

swig 1.1p5
Python 1.5.1
SGI-Irix 6.5/MIPspro 7.2.1

Hello,

I have built a shared module that reads flat files for environment
variable information, and makes subsequent putenv calls for those
pertinent ones. This functionality is encapsulated in a c++ class,
FileRegistry, which I think I have working correctly, using shadow
classes under python. All I expose in the class is the ctor, the dtor,
and the method called exportEnvVars():

%module MDIreg
%{
#include "fileregistry.h"
%}

class FileRegistry
{
public:
FileRegistry(bool, bool);
~FileRegistry();
void exportEnvVars();
};

...

However, I can't seem to get the envvars to persist into the current
process space. If I fork off a child, then I can see the envvars that I
expect. Here is a pretty explanatory log:

[rceci_latest]{rcecil}/vobs/srg/src/registry> python
Python 1.5.1 (#3, May 21 1999, 16:24:24) [C] on irix646-32-mips2
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> import os
>>> import MDIreg
>>> m = MDIreg.FileRegistry(1,1)
>>> m.exportEnvVars()
>>> os.system("env | grep MDI_REG_VERSION")
MDI_REG_VERSION=10.0
0
>>> os.environ['MDI_REG_VERSION']
Traceback (innermost last):
File "<stdin>", line 1, in ?
File "/usr/local/lib/python1.5/UserDict.py", line 12, in __getitem__
def __getitem__(self, key): return self.data[key]
KeyError: MDI_REG_VERSION
>>> os.environ.keys()
[.'WINEDITOR', 'TS', 'MAIL', 'NOMSGLABEL', 'PWD', 'MU',
'MDI_PRINT_UCO_SIZ', 'topdir', 'MDI_ACAR_SITE', 'VOB_ROOT', 'QTDIR',
'LM_LICE
NSE_FILE', 'NPX_PLUGIN_PATH', 'PRINTER', 'HOST', 'PYTHONPATH',
'CLASSPATH', 'MOZILLA_HOME', 'REMOTEUSER', 'NOMSGSEVERITY', 'LIB_TOOL
S', 'CVSROOT', 'SHELL', 'TCLLIBPATH', 'LIB_VAR', 'MANPATH', 'EDITOR',
'CLEARCASE_BLD_UMASK', 'HZ', 'HOME', 'CLEARCASE_ROOT', 'SHLVL'
, 'LIBRARY_PATH', 'TZ', 'BUILDMEISTER', 'TERM', 'LANG', 'VOB_TOP',
'LD_LIBRARY_PATH', 'MSGVERB', 'MAGIC_PATH', 'CVINSTRLIB', 'PATH',
'T', 'USER', 'LOGNAME', 'CPLUS_INCLUDE_PATH', 'SCRIPTS_DIR',
'HOSTTYPE', 'REMOTEHOST', 'BL', 'CLEARCASE_CMDLINE', 'LIB_ARCH']
>>>
>>> os.system("/usr/local/bin/python")
Python 1.5.1 (#3, May 21 1999, 16:24:24) [C] on irix646-32-mips2
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> import os
>>> os.environ['MDI_REG_VERSION']
'10.0'
>>> ^D
0
>>> ^D

---

As you can see I can fork of either a simple csh command, or a
completely new python interpreter, and either way, the children can see
the result of the exportEnvVars() method call. By the way, when I
build my module in debug mode, I can see my "destroyed" messages I
specifically coded in the dtor of the class, to verify that my
FileRegistry object is being destroyed when I expect it.

Any ideas or suggestions?

Thanks

Rob Cecil

Sent via Deja.com http://www.deja.com/
Share what you know. Learn what you don't.

Question about envvars [ In reply to ]

sjturner at ix

Jun 7, 1999, 10:15 AM

Post #2 of 7 (1081 views)

Permalink

Hi Rob,

> I have built a shared module that reads flat files for environment
> variable information, and makes subsequent putenv calls for those
> pertinent ones. This functionality is encapsulated in a c++ class,
> FileRegistry, which I think I have working correctly, using shadow
> classes under python. All I expose in the class is the ctor, the dtor,
> and the method called exportEnvVars():
>
> %module MDIreg
> %{
> #include "fileregistry.h"
> %}
>
> class FileRegistry
> {
> public:
> FileRegistry(bool, bool);
> ~FileRegistry();
> void exportEnvVars();
> };
>
> ...
>
> However, I can't seem to get the envvars to persist into the current
> process space. If I fork off a child, then I can see the envvars that I
> expect.

The problem is that, although putenv does indeed modify the current
process's environment, Python only reads the environment when the posix
module (imported by os) is first imported, so as to create the environ
dictionary. (Try confirming this behavior by postponing "import os"
until after your call to MDIreg.exportEnvVars. You should see your
environment changes reflected in os.environ.)

The proper solution is, I think, to avoid the direct use of the putenv
function, and instead to use the Python C API to modify the os.environ
mapping (which calls putenv as a side effect). Here is a crude C module
to do just such a thing:

-------------------- START modenv.c --------------------
#include <Python.h>

#include <stdio.h>
#include <string.h>

static PyObject *modenv_load(PyObject *self, PyObject *args)
{
FILE *fp;
PyObject *os, *environ, *v;
char buf[256];
char *file;
char *value;

if (!PyArg_ParseTuple(args, "s", &file))
return NULL;

os = PyImport_ImportModule("os");
environ = PyObject_GetAttrString(os, "environ");

if (fp = fopen(file, "r")) {
while (fgets(buf, sizeof(buf), fp), !feof(fp)) {
/* skip lines that do not contain equals sign */
if (!(value = strchr(buf, '=')))
continue;

/* remove trailing newline */
buf[strlen(buf)-1] = '\0';

*value++ = '\0';
v = PyString_FromString(value);
PyMapping_SetItemString(environ, buf, v);
Py_DECREF(v);
}
fclose(fp);
}

Py_DECREF(environ);
Py_DECREF(os);

Py_INCREF(Py_None);
return Py_None;
}

static struct PyMethodDef modenv_methods[] = {
{"load", (PyCFunction)modenv_load, METH_VARARGS},
{NULL, (PyCFunction)NULL, 0, NULL}
};

void initmodenv()
{
if (!Py_InitModule4("modenv", modenv_methods, NULL, NULL,
PYTHON_API_VERSION))
Py_FatalError("can't initialize module modenv");
}
-------------------- END modenv.c --------------------

Hope it helps.

Regards,
Stephen

--
Stephen J. Turner <sjturner@ix.netcom.com>

Question about envvars [ In reply to ]

rceci at my-deja

Jun 7, 1999, 12:50 PM

Post #3 of 7 (1082 views)

Permalink

Stephen,

Boy I was excited to get a reasonable explaination so quickly, but alas,
it still does not work. Here is my test script:

#!/usr/local/bin/python

import MDIreg

reg = MDIreg.FileRegistry(1,1)
reg.exportEnvVars()

import os
print "child has it : "
os.system("env | grep MDI_REG_VERSION")
print "but do I have it? :"
print os.environ['MDI_REG_VERSION']

---

I am interested in this "MDI_REG_VERSION" envvar which gets exported by
my library. Here is the output:

child has it :
MDI_REG_VERSION=10.0
but do I have it? :
Traceback (innermost last):
File "regtest.py", line 12, in ?
print os.environ['MDI_REG_VERSION']
File "/usr/local/lib/python1.5/UserDict.py", line 12, in __getitem__
def __getitem__(self, key): return self.data[key]
KeyError: MDI_REG_VERSION

Any other ideas? Any way to tell the os/posix module to "refresh"?

By the way, thanks for the library implementation suggestion, but I must
keep it the way it is, because other c, and c++ (at the class level)
programs access this library.

Thanks

Rob

In article <375BFE38.3FB3E83C@ix.netcom.com>,
"Stephen J. Turner" <sjturner@ix.netcom.com> wrote:
> Hi Rob,
>
> > I have built a shared module that reads flat files for environment
> > variable information, and makes subsequent putenv calls for those
> > pertinent ones. This functionality is encapsulated in a c++ class,
> > FileRegistry, which I think I have working correctly, using shadow
> > classes under python. All I expose in the class is the ctor, the
dtor,
> > and the method called exportEnvVars():
> >
> > %module MDIreg
> > %{
> > #include "fileregistry.h"
> > %}
> >
> > class FileRegistry
> > {
> > public:
> > FileRegistry(bool, bool);
> > ~FileRegistry();
> > void exportEnvVars();
> > };
> >
> > ...
> >
> > However, I can't seem to get the envvars to persist into the current
> > process space. If I fork off a child, then I can see the envvars
that I
> > expect.
>
> The problem is that, although putenv does indeed modify the current
> process's environment, Python only reads the environment when the
posix
> module (imported by os) is first imported, so as to create the environ
> dictionary. (Try confirming this behavior by postponing "import os"
> until after your call to MDIreg.exportEnvVars. You should see your
> environment changes reflected in os.environ.)
>
> The proper solution is, I think, to avoid the direct use of the putenv
> function, and instead to use the Python C API to modify the os.environ
> mapping (which calls putenv as a side effect). Here is a crude C
module
> to do just such a thing:
>
> -------------------- START modenv.c --------------------
> #include <Python.h>
>
> #include <stdio.h>
> #include <string.h>
>
> static PyObject *modenv_load(PyObject *self, PyObject *args)
> {
> FILE *fp;
> PyObject *os, *environ, *v;
> char buf[256];
> char *file;
> char *value;
>
> if (!PyArg_ParseTuple(args, "s", &file))
> return NULL;
>
> os = PyImport_ImportModule("os");
> environ = PyObject_GetAttrString(os, "environ");
>
> if (fp = fopen(file, "r")) {
> while (fgets(buf, sizeof(buf), fp), !feof(fp)) {
> /* skip lines that do not contain equals sign */
> if (!(value = strchr(buf, '=')))
> continue;
>
> /* remove trailing newline */
> buf[strlen(buf)-1] = '\0';
>
> *value++ = '\0';
> v = PyString_FromString(value);
> PyMapping_SetItemString(environ, buf, v);
> Py_DECREF(v);
> }
> fclose(fp);
> }
>
> Py_DECREF(environ);
> Py_DECREF(os);
>
> Py_INCREF(Py_None);
> return Py_None;
> }
>
> static struct PyMethodDef modenv_methods[] = {
> {"load", (PyCFunction)modenv_load, METH_VARARGS},
> {NULL, (PyCFunction)NULL, 0, NULL}
> };
>
> void initmodenv()
> {
> if (!Py_InitModule4("modenv", modenv_methods, NULL, NULL,
> PYTHON_API_VERSION))
> Py_FatalError("can't initialize module modenv");
> }
> -------------------- END modenv.c --------------------
>
> Hope it helps.
>
> Regards,
> Stephen
>
> --
> Stephen J. Turner <sjturner@ix.netcom.com>
>

Sent via Deja.com http://www.deja.com/
Share what you know. Learn what you don't.

Question about envvars [ In reply to ]

sjturner at ix

Jun 7, 1999, 2:24 PM

Post #4 of 7 (1082 views)

Permalink

Hi Rob,

> Boy I was excited to get a reasonable explaination so quickly, but alas,
> it still does not work. Here is my test script:
>
> #!/usr/local/bin/python
>
> import MDIreg
>
> reg = MDIreg.FileRegistry(1,1)
> reg.exportEnvVars()
>
> import os
> print "child has it : "
> os.system("env | grep MDI_REG_VERSION")
> print "but do I have it? :"
> print os.environ['MDI_REG_VERSION']
>
> ---
>
> I am interested in this "MDI_REG_VERSION" envvar which gets exported by
> my library. Here is the output:
>
> child has it :
> MDI_REG_VERSION=10.0
> but do I have it? :
> Traceback (innermost last):
> File "regtest.py", line 12, in ?
> print os.environ['MDI_REG_VERSION']
> File "/usr/local/lib/python1.5/UserDict.py", line 12, in __getitem__
> def __getitem__(self, key): return self.data[key]
> KeyError: MDI_REG_VERSION

Oops! I just realized that the site module, which is imported by
default when the Python interpreter starts, imports the os module -- so
by the time your script begins to execute, it's already too late. Just
for grins, you could try disabling the site module by adding the '-S'
option to the Python command line, as in:

#!/usr/local/bin/python -S

I'm not advocating this approach, since it's generally a good idea to
import the site module. I'd just like to know if it works for you.

> Any other ideas? Any way to tell the os/posix module to "refresh"?

None that I can see from looking in Modules/posixmodule.c. Sorry.

> By the way, thanks for the library implementation suggestion, but I must
> keep it the way it is, because other c, and c++ (at the class level)
> programs access this library.

OK, how 'bout this? Suppose you added a virtual "putenv" method to your
FileRegistry C++ class whose default implementation was to call the
global putenv function. Then in your Python C extension, you could
subclass FileRegistry and override the putenv method with one that
modifies os.environ as described in the prior post.

Regards,
Stephen

--
Stephen J. Turner <sjturner@ix.netcom.com>

Question about envvars [ In reply to ]

rceci at my-deja

Jun 7, 1999, 4:08 PM

Post #5 of 7 (1079 views)

Permalink

Stephen,

Yes -S does work!. Being rather new to python, what exactly do I lose
if I specify that switch ? What is in the site module? Can I
compensate for the -S with manual "import <module>" commands in my
script?

In article <375C3891.1D11E1A4@ix.netcom.com>,
"Stephen J. Turner" <sjturner@ix.netcom.com> wrote:
> Hi Rob,
>
> > Boy I was excited to get a reasonable explaination so quickly, but
alas,
> > it still does not work. Here is my test script:
> >
> > #!/usr/local/bin/python
> >
> > import MDIreg
> >
> > reg = MDIreg.FileRegistry(1,1)
> > reg.exportEnvVars()
> >
> > import os
> > print "child has it : "
> > os.system("env | grep MDI_REG_VERSION")
> > print "but do I have it? :"
> > print os.environ['MDI_REG_VERSION']
> >
> > ---
> >
> > I am interested in this "MDI_REG_VERSION" envvar which gets exported
by
> > my library. Here is the output:
> >
> > child has it :
> > MDI_REG_VERSION=10.0
> > but do I have it? :
> > Traceback (innermost last):
> > File "regtest.py", line 12, in ?
> > print os.environ['MDI_REG_VERSION']
> > File "/usr/local/lib/python1.5/UserDict.py", line 12, in
__getitem__
> > def __getitem__(self, key): return self.data[key]
> > KeyError: MDI_REG_VERSION
>
> Oops! I just realized that the site module, which is imported by
> default when the Python interpreter starts, imports the os module --
so
> by the time your script begins to execute, it's already too late.
Just
> for grins, you could try disabling the site module by adding the '-S'
> option to the Python command line, as in:
>
> #!/usr/local/bin/python -S
>
> I'm not advocating this approach, since it's generally a good idea to
> import the site module. I'd just like to know if it works for you.
>
> > Any other ideas? Any way to tell the os/posix module to "refresh"?
>
> None that I can see from looking in Modules/posixmodule.c. Sorry.
>
> > By the way, thanks for the library implementation suggestion, but I
must
> > keep it the way it is, because other c, and c++ (at the class level)
> > programs access this library.
>
> OK, how 'bout this? Suppose you added a virtual "putenv" method to
your
> FileRegistry C++ class whose default implementation was to call the
> global putenv function. Then in your Python C extension, you could
> subclass FileRegistry and override the putenv method with one that
> modifies os.environ as described in the prior post.
>
> Regards,
> Stephen
>
> --
> Stephen J. Turner <sjturner@ix.netcom.com>
>

Sent via Deja.com http://www.deja.com/
Share what you know. Learn what you don't.

Question about envvars [ In reply to ]

rceci at my-deja

Jun 7, 1999, 8:06 PM

Post #6 of 7 (1092 views)

Permalink

Another thought,

If I specify a -S, can I compensate by specifically saying at the
appropriate time, "import site" ?

Rob

In article <375C3891.1D11E1A4@ix.netcom.com>,
"Stephen J. Turner" <sjturner@ix.netcom.com> wrote:
> Hi Rob,
>
> > Boy I was excited to get a reasonable explaination so quickly, but
alas,
> > it still does not work. Here is my test script:
> >
> > #!/usr/local/bin/python
> >
> > import MDIreg
> >
> > reg = MDIreg.FileRegistry(1,1)
> > reg.exportEnvVars()
> >
> > import os
> > print "child has it : "
> > os.system("env | grep MDI_REG_VERSION")
> > print "but do I have it? :"
> > print os.environ['MDI_REG_VERSION']
> >
> > ---
> >
> > I am interested in this "MDI_REG_VERSION" envvar which gets exported
by
> > my library. Here is the output:
> >
> > child has it :
> > MDI_REG_VERSION=10.0
> > but do I have it? :
> > Traceback (innermost last):
> > File "regtest.py", line 12, in ?
> > print os.environ['MDI_REG_VERSION']
> > File "/usr/local/lib/python1.5/UserDict.py", line 12, in
__getitem__
> > def __getitem__(self, key): return self.data[key]
> > KeyError: MDI_REG_VERSION
>
> Oops! I just realized that the site module, which is imported by
> default when the Python interpreter starts, imports the os module --
so
> by the time your script begins to execute, it's already too late.
Just
> for grins, you could try disabling the site module by adding the '-S'
> option to the Python command line, as in:
>
> #!/usr/local/bin/python -S
>
> I'm not advocating this approach, since it's generally a good idea to
> import the site module. I'd just like to know if it works for you.
>
> > Any other ideas? Any way to tell the os/posix module to "refresh"?
>
> None that I can see from looking in Modules/posixmodule.c. Sorry.
>
> > By the way, thanks for the library implementation suggestion, but I
must
> > keep it the way it is, because other c, and c++ (at the class level)
> > programs access this library.
>
> OK, how 'bout this? Suppose you added a virtual "putenv" method to
your
> FileRegistry C++ class whose default implementation was to call the
> global putenv function. Then in your Python C extension, you could
> subclass FileRegistry and override the putenv method with one that
> modifies os.environ as described in the prior post.
>
> Regards,
> Stephen
>
> --
> Stephen J. Turner <sjturner@ix.netcom.com>
>

Sent via Deja.com http://www.deja.com/
Share what you know. Learn what you don't.

Question about envvars [ In reply to ]

sjturner at ix

Jun 8, 1999, 6:17 AM

Post #7 of 7 (1080 views)

Permalink

Rob <rceci@my-deja.com> wrote:
> If I specify a -S, can I compensate by specifically saying at the
> appropriate time, "import site" ?

Yes you can. In fact, older versions of Python required this; the
implicit "import site" on interpreter startup was added as of 1.5a4.
For full information on the site module, refer to the documentation at:
http://www.python.org/doc/current/lib/module-site.html

However, if you plan to go with this approach, consider that you
probably won't want to install your MDIreg extension module in the
$exec_prefix/lib/python1.5/site-packages directory (the usual place to
install site-wide extensions), since that directory is added to sys.path
by the site module. You'll more likely have to put it in a custom
directory and make sure that directory in named in the PYTHONPATH
environment variable.

Even so, I'd be concerned about the fragility introduced into your
MDIreg module by this approach -- namely, that the
FileRegistry.exportEnvVars method cannot be used once the os module is
imported. I'd much prefer to see you try the C++ subclassing suggested
below, so that your extension module works regardless of the module
import order.

Regards,
Stephen

> In article <375C3891.1D11E1A4@ix.netcom.com>,
> "Stephen J. Turner" <sjturner@ix.netcom.com> wrote:
>> Hi Rob,
>>
>>> Boy I was excited to get a reasonable explaination so quickly, but
>>> alas, it still does not work. Here is my test script:
>>>
>>> #!/usr/local/bin/python
>>>
>>> import MDIreg
>>>
>>> reg = MDIreg.FileRegistry(1,1)
>>> reg.exportEnvVars()
>>>
>>> import os
>>> print "child has it : "
>>> os.system("env | grep MDI_REG_VERSION")
>>> print "but do I have it? :"
>>> print os.environ['MDI_REG_VERSION']
>>>
>>> ---
>>>
>>> I am interested in this "MDI_REG_VERSION" envvar which gets exported
>>> by my library. Here is the output:
>>>
>>> child has it :
>>> MDI_REG_VERSION=10.0
>>> but do I have it? :
>>> Traceback (innermost last):
>>> File "regtest.py", line 12, in ?
>>> print os.environ['MDI_REG_VERSION']
>>> File "/usr/local/lib/python1.5/UserDict.py", line 12, in __getitem__
>>> def __getitem__(self, key): return self.data[key]
>>> KeyError: MDI_REG_VERSION
>>
>> Oops! I just realized that the site module, which is imported by
>> default when the Python interpreter starts, imports the os module --
>> so by the time your script begins to execute, it's already too late.
>> Just for grins, you could try disabling the site module by adding the
>> '-S' option to the Python command line, as in:
>>
>> #!/usr/local/bin/python -S
>>
>> I'm not advocating this approach, since it's generally a good idea to
>> import the site module. I'd just like to know if it works for you.
>>
>>> Any other ideas? Any way to tell the os/posix module to "refresh"?
>>
>> None that I can see from looking in Modules/posixmodule.c. Sorry.
>>
>>> By the way, thanks for the library implementation suggestion, but I
>>> must keep it the way it is, because other c, and c++ (at the class
>>> level) programs access this library.
>>
>> OK, how 'bout this? Suppose you added a virtual "putenv" method to
>> your FileRegistry C++ class whose default implementation was to call
>> the global putenv function. Then in your Python C extension, you
>> could subclass FileRegistry and override the putenv method with one
>> that modifies os.environ as described in the prior post.

--
Stephen J. Turner <sjturner@ix.netcom.com>