Mailing List Archive

Chris Williamson discovers that:

> Py_Initialize();
> Py_Finalize();

leaks.

Well, if MSVC reported on all of MFC's static allocations, you'd see
that it leaks, too.

These are nothing to worry about. It's static data that Python needs
to run. It won't continually grow. If you use the debug memory stuff,
you should take your checkpoint after Py_Initialize and report before
Py_Finalize. In fact, you should probably take your checkpoint after
you've done your imports, because a lot of the memory they use won't
get released until Py_Finalize.



- Gordon

This, however, is going to be a giant problem for me.

I'm porting Python to Novell NetWare, and that platform complains quite a
bit when a process is unloaded without freeing up it's memory. Also, SWIG
produces wrappers that suck up a lot of memory in init.

Because Python doesn't finalize modules, I can't clean up this memory...

I'm going to have to write my own memory allocator and fake out Python and
all add-on modules to use it, what a major pain.


--
Brad Clements,
bkc@murkworks.com
Gordon McMillan <gmcm@hypernet.com> wrote in message
news:1279335510-28879097@hypernet.com...
> Chris Williamson discovers that:
>
> > Py_Initialize();
> > Py_Finalize();
>
> leaks.
>
> Well, if MSVC reported on all of MFC's static allocations, you'd see
> that it leaks, too.
>
> These are nothing to worry about. It's static data that Python needs
> to run. It won't continually grow. If you use the debug memory stuff,
> you should take your checkpoint after Py_Initialize and report before
> Py_Finalize. In fact, you should probably take your checkpoint after
> you've done your imports, because a lot of the memory they use won't
> get released until Py_Finalize.
>
>
>
> - Gordon
>
>

Brad Clements wrote in message <7nclk9$sg1$1@news.clarkson.edu>...

>This, however, is going to be a giant problem for me.

Its a little problem for a few of us too.

>Because Python doesn't finalize modules, I can't clean up this memory...

This has been discussed before, and there seems to be general agreement
Python _should_ take some steps to finalize modules.

Further, although PyInit/PyFinalize leaks "static" memory, I believe
(although have not confirmed) that repeatedly calling this pair _will_
leak - although it is uncommon for apps to need to do this, it can happen.

>I'm going to have to write my own memory allocator and fake out Python and
>all add-on modules to use it, what a major pain.

Well, I would like to suggest that you help with getting Python to clear
this stuff up. I dont think there is anything seriously hard about the
cleanup, other than a general feeling that this memory is "not a problem".
Still quite a pain, but then your efforts help mankind :-)

I dont think anyone would complain about a Python that freed up every single
memory allocation it made. Someone just has to do it.

Re the module finalization: Maybe a proposal could be put forward for a
module to optionally register a "module term" function (by passing a pointer
to a function at module init time - Python could stash this away somewhere,
calling it as the module is unloaded). This would be backward compatible,
and allow modules to cleanup all their allocations - something that is
non-trivial and hacky for an extension module to do today.

You would probably find a number of people would be interested in this, and
be willing to help out with the tricky stuff once you (or someone!) takes
the lead. I know Pythonwin and PythonCOM could benefit, and believe from
recent discussions wxPython could too. Im sure the list is quite large.
Unfortunately for you, it seems you are the only one for which it is a
"giant problem"

Mark.

Heh, ain't that always the case.

On another note, config.h and friends really need a cleanup. The nested
ifdefs are very difficult to figure out.

What's the *best* way for me to address this? I'd hate to go through the
1.5.2 source cleaning this up, just to see all the work lost.

Or worse, *think* I'm cleaning it up, but breaking it for Unix folks.

I'd like to:

1. Cleanup config.h and get all configuration options there. Some are hiding
out in posix and socket module too
2. Encapsulate all memory alloc/free for all modules through a set of
"My_Malloc" type calls that could be #def as must malloc/free or something
else depending on the platform.
3. Module finalization.

Not having worked on group projects of this size, I'm not sure what the best
method is to address these and make sure the changes aren't lost (and that
the changes are acceptable)

One other thing ... The current config.h implies that a given compiler
defines the target environment, which isn't true. I"m using Borland C as a
cross compiler for Netware (MSC can do it too), so I have to separate the
run-time environment options from the compiler selection. Thats the first
thing I want to address.


--
Brad Clements,
bkc@murkworks.com
Mark Hammond <MHammond@skippinet.com.au> wrote in message
news:7nitl6$q2r$1@m2.c2.telstra-mm.net.au...
> Brad Clements wrote in message <7nclk9$sg1$1@news.clarkson.edu>...
>
> Im sure the list is quite large.
> Unfortunately for you, it seems you are the only one for which it is a
> "giant problem"
>