Mailing List Archive

* Heather A. Drury
|
| Any hints/suggestions/pointers would be greatly appreciated.

One easy way is to get my plumbo module and see if it can find any
cycles in your program. It only understands what happens in
Python-space and it's not complete, but several people have found it
useful.

<URL: http://www.stud.ifi.uio.no/~larsga/download/python/plumbo.py >


If Plumbo doesn't solve your problem (or is too slow), you can try Tim
Peter's more complex, complete and faster Cyclops, which he released
on Monday.

<URL: http://www.python.org/ftp/python/contrib/System/Cyclops.py >

--Lars M.

On 21 Jul 1999 11:31:27 -0500, Heather A. Drury <heather@thalamus.wustl.edu> wrote:
>Hi,
>
>I'm relatively new to python and am using it (and Tkinter,
>VTK, NumPy, numpyio, and some other modules that I've probably
>forgotten) to write a volume/slice visualization package.
>
>Here's what I see on a SGI Indigo2 Max Impact with 448 Meg of memory:
>
> before I start application 350 meg free
> start application 342
> read volume 1 284
> re-read volume 1 229
> re-read volume 1 192
> re-read volume 1 173
> re-read volume 1 155


Are you storing data as textures, or display lists?
If VTK is an interface to OpenGL, then you need to clean up
some resources. In my program, if I create a large number
of display lists with lots of geometry or texture data, they
take up a lot of RAM.

I haven't used VTK with python, only PyOpenGL.

Dave
--
--------------------------------------------------------------------------------
Email: dek@cgl.ucsf.edu David Konerding WWW: http://picasso.ucsf.edu/~dek
--------------------------------------------------------------------------------
Snail: Graduate Group in Biophysics
Medical Sciences 926, Box 0446
University of California
San Francisco, CA 94143

In article <7n4skv$8do@thalamus.wustl.edu>,
Heather A. Drury <heather@thalamus.wustl.edu> wrote:
>
>The program has gotten to a fairly advanced stage but I've run
>into what appears to be a major memory leak. I come
>from the C programming world and have previously used
>Purify to debug memory leaks. Alas, I thought using python/vtk
>would take care of all this memory garbage for me (although
>I'm probably not doing something I should be).
>
>Over time as the user continues to use the application, the
>computer starts using up swap space and eventually the program
>crashes. I haven't the faintest idea how to begin to debug this
>problem. I'm sure I need to provide some more detail but the
>code is about 2000 lines now and I'm not sure what detail
>would be useful.
>
>Any hints/suggestions/pointers would be greatly appreciated.

This sounds like a problem an awful lot like I had with NumPy
a few weeks ago. Basically, slices of arrays count as references
to it, which leads to some surprising memory wastage if you
don't know what's happening.

For example:

import Numeric

def waste_memory():
x = Numeric.zeros([1000,1000])
return x[0,0:2]

Each call to waste_memory() will generate a million-element array,
and return an array of two elements. However, the slice that's
returned will cause the *whole* 1000x1000 array to be kept in
memory.

You can fix this by wrapping the return in another call to
Numeric.array(), which creates a new array object and lets
the old one go out of scope properly.

E.g.:

def no_waste():
x = Numeric.zeros([1000,1000])
return Numeric.array( x[0,0:2] )

Hope this helps.


Neel

"Heather A. Drury" wrote:
>
> I thought using python/vtk
> would take care of all this memory garbage for me

A reasonable expectation, but unfortunately Python
doesn't quite deliver. Garbage collection in Python
is by reference counting only, so if there are any
cycles in your garbage, it never goes away.

You have to design your program so that it either
doesn't create any cycles or ensures that they are
broken when the data is no longer needed.

There was a module called Cyclops announced here
by Tim Peters recently for finding cycles in data
structures. Take a look back in dejanews.

Greg

I posted a module called Debug to track down memory leaks. It requires
specific code be added to your __init__ and __del__ routines, as well as
having a Destroy method we use to eliminate circular references. If you
want it I will post it again.

-----Original Message-----
From: Greg Ewing [mailto:greg.ewing@compaq.com]
Sent: Thursday, July 22, 1999 4:04 PM
To: python-list@cwi.nl
Subject: Re: how do I find memory leaks?


"Heather A. Drury" wrote:
>
> I thought using python/vtk
> would take care of all this memory garbage for me

A reasonable expectation, but unfortunately Python
doesn't quite deliver. Garbage collection in Python
is by reference counting only, so if there are any
cycles in your garbage, it never goes away.

You have to design your program so that it either
doesn't create any cycles or ensures that they are
broken when the data is no longer needed.

There was a module called Cyclops announced here
by Tim Peters recently for finding cycles in data
structures. Take a look back in dejanews.

Greg

"Stidolph, David" <stidolph@origin.ea.com> writes:

> I posted a module called Debug to track down memory leaks. It requires
> specific code be added to your __init__ and __del__ routines, as well as
> having a Destroy method we use to eliminate circular references. If you
> want it I will post it again.

Please do!

Matt Gushee
Portland, Maine, USA

From: Matt Gushee <mgushee@havenrock.com>
Subject: Re: how do I find memory leaks?
Organization: Internet Maine/HarvardNet

"Stidolph, David" <stidolph@origin.ea.com> writes:

> I posted a module called Debug to track down memory leaks. It requires
> specific code be added to your __init__ and __del__ routines, as well as
> having a Destroy method we use to eliminate circular references. If you
> want it I will post it again.

Please do!

Matt Gushee
Portland, Maine, USA
--
|Fidonet: UUCP 2:500/3.1
|Internet: UUCP@p1.f3.n500.z2.hccfido.hcc.nl
|
| Standard disclaimer: The views of this user are strictly his own.

The following is our Debug.py file. We have a Globals.py file that has all
of our global variables, include one called 'debug' that is a class instance
of the Debug class. Note that all base classes should have the following
code in them:

import Globals #for Globals.debug

class Name:

def __del__(self):
if __debug__: #we do this to toss in non-debug situations
Globals.debug.LogDel(self)

def __init__(self):
if __debug__:
Globals.debug.LogInit(self)

def Destroy(self):
if __debug__:
Globals.debug.LogDestroy(self)
#Eliminate pointers to other classes and notify any other classes
that point to it that
#you are going away and need to have references eliminated.


Please notice that all base classes have these routines defined. Classes
derived from them do NOT have to have this code - in fact, it will cause
warning messages. As LogInit is called, we keep the stack trace for its
creation. When LogDestroy is called it is noted, and when LogDel is called
the entry is removed. I set up additional variables to control reporting in
debug mode. The idea of this module is to track class instances and issue a
report at shutdown as to what (if anything) is left undeleted, or didn't
have the Destroy method called - or had the Destroy method called more than
once. Output is to the standard output, which we have routed to
OutputDebugString so it goes to the MSDev Output debug window. Any
questions? Suggestions??

Thanks,

David Stidolph

P.S. On our first use of this we found hundreds of memory leaks. We solved
most within four hours.

#### And now for the Debug Module

#Debug Module
#
#This class is designed to track Python objects between their __init__() and
Destroy() method calls.
#There are different 'levels' to the tracking. If we do it at all, we will
track the objects by the object
#id() using a dictionary. If we have 'holdTraceback' active, then in
LogInit we get the current
#stack, translate it into a list of strings and use that list as the value
and the object reference as
#the key in the dictionary.
#
#As a dictionary we keep only one copy and don't allow for
LogInit/LogDestroy to be called more than
#once (have setting to throw exception if it occurs). This means you put
the calls to LogInit and
#LogDestroy in the base class .. not the derived classes. Proper class name
of the object is used
#by __class__.__name__.
#
#TODO:
# Fix output for MSVC IDE to use absolute paths to Python files for
double-clicking
# (Note that this works now for Windows 95, but not 98)
# Add Python object for output in LogReport to allow output to file or
sending across net
# Add print formatted output to wrap output (allow to send across
net).
# Optimize use of stack trace to reduce memory requirements.

from sys import exc_info
from traceback import extract_stack

class Debug:
"""Only one instance allowed"""

__refDebugCount = 0 # class variable

def __del__(self):
Debug.__refDebugCount = Debug.__refDebugCount - 1

def __init__(self, trackObjects = 1, holdTraceback = 1,
raiseExceptionOnDuplicate = 0, displayInit = 0, displayDestroy = 0):
if Debug.__refDebugCount != 0:
raise "Only one copy of debug allowed"

if __debug__:
self.trackObjects = trackObjects
self.holdTraceback = holdTraceback
self.raiseExceptionOnDuplicate = raiseExceptionOnDuplicate
self.displayInit = displayInit
self.displayDestroy = displayDestroy
else:
self.trackObjects = 0 #if we don't have debug mode, then
it doesn't matter what
self.holdTraceback = 0 #settings we use since nothing
SHOULD be calling us.
self.raiseExceptionOnDuplicate = 0 #Just in case we ignore and set
for nothing.
self.displayInit = 0
self.displayDestroy = 0

self.activeObjects = {} #clear dictionary
Debug.__refDebugCount = 1 #we are initialized

def Destroy(self):
Debug.__refDebugCount = 0
self.activeObjects = None

def LogDel(self,object):
#We wrap the destruction of the key inside a try..except block in case
the LogDestroy is called without
#calling the LogInit (rare) or LogDestroy is being called for a second
time (far more likely).
try:
objectID = id(object)
value = self.activeObjects[objectID]
if 0 == value[1]:
if self.raiseExceptionOnDuplicate:
raise "MEM: LogDel called on "+object.__class__.__name__+" without
calling Destroy()"
else:
print "MEM: LogDel called on "+object.__class__.__name__+" without
calling Destroy()"
if self.holdTraceback:
tb = value[2]
for tuple in tb[:-2]: #-2 is to eliminate report of call to
LogInit()
print str(tuple[0])+'('+str(tuple[1])+'): in
'+str(tuple[2])+','
print ' '+str(tuple[3])
print
del self.activeObjects[objectID]
except:
if self.raiseExceptionOnDuplicate:
raise "MEM: LogDel called twice on "+object.__class__.__name__+".
See if base class doing LogInit/LogDestroy/LogDel (only one should)"
else:
print
print "MEM: LogDel called twice on "+object.__class__.__name__+".
See if base class doing LogInit/LogDestroy/LogDel (only one should)"
print

#Called from each base object Destroy() method.
def LogDestroy(self,object):
if self.trackObjects:
if self.displayDestroy:
print 'MEM: ',repr(object),'of class',object.__class__.__name__,'is
being Destroyed'
#We wrap the destruction of the key inside a try..except block in case
the LogDestroy is called without
#calling the LogInit (rare) or LogDestroy is being called for a second
time (far more likely).
self.activeObjects[id(object)][1] = 1

#Called from base object __init__(self) routines
def LogInit(self,object):
if self.trackObjects:
if self.displayInit:
#display that the object has been constructed and its __init__ has
been called
print 'MEM: ',repr(object),'of class',object.__class__.__name__,'is
being initialized'

#traceback of stack is generated using extract_stack() and used as the
value in the
#dictionary with the object reference used as the key.
if self.holdTraceback:
#We check for the object already being in the dictionary ONLY if we
will throw an
#exception, silent otherwise
if self.raiseExceptionOnDuplicate:
if self.activeObjects.haskey(object):
raise "Object has already called LogInit"
tb = extract_stack()
self.activeObjects[id(object)] = [object.__class__.__name__,0,tb]
else:
self.activeObjects[id(object)] = [object.__class__.__name__,0,1]

#Called at program end to report on undestroyed Python objects.
def LogReport(self):
# For formatting that Visual Studio can recognize, we're adjusting
# output to look like:
# c:\test\gagob.cpp(236) : error C2065: 'Clear' : undeclared identifier
if self.trackObjects:
if len(self.activeObjects) > 0:
print
print 'MEM: UnDestroyed Python Objects Report:'
print ' %s python objects total.' % len(self.activeObjects)
print
print 'Summary: These objects were not released. Individual stack
traces follow'
for objectID,value in self.activeObjects.items():
print ' class ' + value[0]
print
print
#Get Object reference and stack traceback for each undestroyed
Python object
for objectID,value in self.activeObjects.items():
className = value[0]
destroyCalled = value[1]
tb = value[2]
if destroyCalled == 1:
print 'MEM: class ' + className + ' Destroy() called, but not
__del__.'
else:
print 'MEM: class ' + className + ' Destroy() and __del__
methods not called.'
if self.holdTraceback:
for tuple in tb[:-2]: #-2 is to eliminate report of call to
LogInit()
print str(tuple[0])+'('+str(tuple[1])+'): in
'+str(tuple[2])+','
print ' '+str(tuple[3])
print #blank line between
else:
print 'MEM: Congratulations! No Python object leaks!'


def PrintStack():
tb = extract_stack()
for tuple in tb[:-2]: #-2 is to eliminate report of call to PrintStack()
print str(tuple[0])+'('+str(tuple[1])+'): in '+str(tuple[2])+','
print ' '+str(tuple[3])










-----Original Message-----
From: Matt Gushee [mailto:mgushee@havenrock.com]
Sent: Thursday, July 22, 1999 7:54 PM
To: python-list@cwi.nl
Subject: Re: how do I find memory leaks?


"Stidolph, David" <stidolph@origin.ea.com> writes:

> I posted a module called Debug to track down memory leaks. It requires
> specific code be added to your __init__ and __del__ routines, as well as
> having a Destroy method we use to eliminate circular references. If you
> want it I will post it again.

Please do!

Matt Gushee
Portland, Maine, USA

"Stidolph, David" <stidolph@origin.ea.com> writes:

> The following is our Debug.py file.

Thanks!