Mailing List Archive: Memory leaks in 5.002beta1?

Memory leaks in 5.002beta1?

aburlison at cix

Dec 7, 1995, 12:42 AM

Post #1 of 9 (420 views)

I have a long-running perl script which is used to process a 28Mb file.
This makes extensive use of objects and complex nested reference
datatypes, but other than that doesn't do anything too clever, eg no
evals etc. Unfortunately it leaks memory at a steady rate - it gets to
well over 30Mb before even getting half way through the file. It is
difficult/impossible for me to produce a simple version of the program.
I have instrumented the new and DESTROY methods of the appropriate
classes, and everything seems to match up.

Can anyone suggest any approaches I might take to track this down? Would
explicitly using undef() to tear down all the nested hashes & arrays that
I have created help?

Alan Burlison aburlison@cix.compulink.co.uk

Re: Memory leaks in 5.002beta1? [ In reply to ]

Tim.Bunce at ig

Dec 7, 1995, 1:30 AM

Post #2 of 9 (419 views)

Re: Memory leaks in 5.002beta1? [ In reply to ]

Tim.Bunce at ig

Dec 7, 1995, 1:30 AM

Post #3 of 9 (420 views)

Re: Memory leaks in 5.002beta1? [ In reply to ]

Tim.Bunce at ig

Dec 7, 1995, 1:30 AM

Post #4 of 9 (419 views)

Re: Memory leaks in 5.002beta1? [ In reply to ]

Tim.Bunce at ig

Dec 7, 1995, 1:30 AM

Post #5 of 9 (418 views)

Re: Memory leaks in 5.002beta1? [ In reply to ]

Dec 7, 1995, 9:14 AM

Post #6 of 9 (415 views)

On Thu, 07 Dec 1995 00:42:00 GMT, Alan Burlison wrote:
>I have a long-running perl script which is used to process a 28Mb file.
>This makes extensive use of objects and complex nested reference
>datatypes, but other than that doesn't do anything too clever, eg no
>evals etc. Unfortunately it leaks memory at a steady rate - it gets to
>well over 30Mb before even getting half way through the file. It is
>difficult/impossible for me to produce a simple version of the program.
>I have instrumented the new and DESTROY methods of the appropriate
>classes, and everything seems to match up.
>
>Can anyone suggest any approaches I might take to track this down? Would
>explicitly using undef() to tear down all the nested hashes & arrays that
>I have created help?
>

If you're getting into nested refs, be very careful not to reference
a ref within itself (or if you do, explicitly break the cycle when you
don't need the ref). This trap is easy to fall into, especially in
complex multilevel data. For a silly example:

while (1) {
my $a = [];
$a->[0] = $a;
# $a->[0] = ""; # the "fix": break the cycle
}

- Sarathy.
gsar@engin.umich.edu

Re: Memory leaks in 5.002beta1? [ In reply to ]

aburlison at cix

Dec 7, 1995, 10:12 AM

Post #7 of 9 (418 views)

In-Reply-To: <9512070830.AA07108@toad.ig.co.uk>
Tim,

> I've been meaning to do this patch for awhile. This seems like a very
> good excuse!

I am going to roll back to 5.001m first, and see if that also has the
problem (I don't think it does). I'll then apply your patch to beta2
patch level d & see how it goes, probably some time over the weekend
(unfortunately, real work beckons!)

Alan Burlison aburlison@cix.compulink.co.uk

Re: Memory leaks in 5.002beta1? [ In reply to ]

Dec 7, 1995, 11:29 AM

Post #8 of 9 (418 views)

Tim Bunce wrote :
|| I've been meaning to do this patch for awhile. This seems like a very good
|| excuse!
||
|| Building perl with -DDEBUGGING enables the expensive malloc.c RCHECK code
|| but does not enable the much cheaper MSTATS code (presumably because it
|| wasn't much use without adding code to call the mstats dump function).
||
|| This patch lets -DDEBUGGING also enable the MSTATS code and perl will now
|| automatically call the mstats dump function at the end of perl_run() if the
|| environment variable PERL_DEBUG_MSTATS is set. As an added bonus if it's
|| set >= 2 then it'll also dump the stats at the end of perl_parse so you
|| can separate and compare the memory cost of compilation vs execution.

This looks really useful. It might be nice to toss in an
extension with subroutine that takes an int and a string and
then if (the int xor PERL_DEBUG_MSTATS) is non-zero then the
mstat subroutine gets called with the string srgument provided
as a title.

|| p.s. You'll notice that nothing ever gets allocated into the 8 byte bucket.
|| Here's why:

This looks (at my quick first glance) as if it *always* goes one
bucket too high, for all sizes. If so, then there is a big
space saving here.

--
Maybe we can fix reality one of these days. | John Macdonald
<- Larry Wall Carl Dichter -> | jmm@Elegant.COM
I'd be happy if we could just index it. |

Re: Memory leaks in 5.002beta1? [ In reply to ]

Tim.Bunce at ig

Dec 8, 1995, 4:38 AM

Post #9 of 9 (416 views)

> From: jmm@elegant.com (John Macdonald)
>
> Tim Bunce wrote :
> || I've been meaning to do this patch for awhile. This seems like a very good
> || excuse!
> ||
> || Building perl with -DDEBUGGING enables the expensive malloc.c RCHECK code
> || but does not enable the much cheaper MSTATS code (presumably because it
> || wasn't much use without adding code to call the mstats dump function).
> ||
> || This patch lets -DDEBUGGING also enable the MSTATS code and perl will now
> || automatically call the mstats dump function at the end of perl_run() if the
> || environment variable PERL_DEBUG_MSTATS is set. As an added bonus if it's
> || set >= 2 then it'll also dump the stats at the end of perl_parse so you
> || can separate and compare the memory cost of compilation vs execution.
>
> This looks really useful.

Thanks. I spent way too long doing it when I should have been doing real work!

> It might be nice to toss in an
> extension with subroutine that takes an int and a string and
> then if (the int xor PERL_DEBUG_MSTATS) is non-zero then the
> mstat subroutine gets called with the string srgument provided
> as a title.

Perhaps. I arranged for dump_mstats() to always be defined so that
extensions can call it if desired.

But note that the widespread use of internal caches means that it would
be difficult to draw meaningful conclusions from stats dumped while
the code is running.

Some code to return the free and used space in the various arenas would
be very useful.

> || p.s. You'll notice that nothing ever gets allocated into the 8 byte bucket.
> || Here's why:
>
> This looks (at my quick first glance) as if it *always* goes one
> bucket too high, for all sizes. If so, then there is a big
> space saving here.

Feel free to investigate. I've no time.

Tim.