Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. Perl>
  3. porters
BUG REPORT: Script that demonstrates a memory leak in 5.002beta1
aburlison at cix
Dec 9, 1995, 9:12 PM
Post #1 of 4 (420 views)
Permalink
In-Reply-To: <memo.55405@cix.compulink.co.uk>
The following short script leaks like a sieve on 5.002beta1, but not at
all on 5.001m. I have included some memory statistics, and the output of
myconfig. Some curious things: The number of items pushed onto the
anonymous array referenced by $leak is critical - any less than 125 and
the program size remains constant. If $leak is changed to @leak, the
problem also disappears. Could someone a) confirm that there is a
problem and b) if so, shed some light on what is causing it?

# The following code remains at a constant size when running under 5.001m,
# but grows continuously under 5.002beta1 patch level f
my $i;
for ($i = 10000; $i > 0; $i--)
{
my $leak = [];
my $j;
# The number 125 is critical -
# any less than this and the program size stays constant
for ($j = 125; $j > 0; $j--) { push(@$leak, "Z"); }
}

MEMORY STATS - Perl compiled with -DDEBUGGING_MSTATS
====================================================
Run 1: $j = 124
---------------
Memory allocation statistics after compilation: (buckets 8..16384)
21368 free: 187 26 54 31 13 5 3 3 4 0 0 0
52360 used: 69 102 202 193 19 3 1 9 1 0 0 1
Memory allocation statistics after execution: (buckets 8..16384)
23352 free: 187 26 180 31 13 5 3 1 4 0 0 0
54472 used: 69 102 204 193 19 3 1 11 1 0 0 1

Run 2: $j = 125
---------------
Memory allocation statistics after compilation: (buckets 8..16384)
21368 free: 187 26 54 31 13 5 3 3 4 0 0 0
52360 used: 69 102 202 193 19 3 1 9 1 0 0 1
Memory allocation statistics after execution: (buckets 8..16384)
23352 free: 187 26 180 31 13 5 3 1 4 0 0 0
5174472 used: 69 102 204 193 19 3 10001 11 1 0 0 1

OUTPUT OF myconfig
==================
Summary of my perl5 (patchlevel 2) configuration:
Platform:
osname=linux, osver=1, archname=i486-linux
uname='linux fubar 1.2.13 #2 sat oct 28 19:55:36 gmt 1995 i486 '
hint=previous
Compiler:
cc='gcc', optimize='-O2', ld='gcc'
cppflags='-D__USE_BSD_SIGNAL -Dbool=char -DHAS_BOOL
-DDEBUGGING_MSTATS'
ccflags ='-D__USE_BSD_SIGNAL -Dbool=char -DHAS_BOOL
-DDEBUGGING_MSTATS'
ldflags =''
stdchar='char', d_stdstdio=define, usevfork=false
voidflags=15, castflags=0, d_casti32=undef, d_castneg=define
intsize=4, alignbytes=4, usemymalloc=y, randbits=31
Libraries:
so=so
libpth=/lib /usr/lib
libs=-lndbm -lgdbm -ldbm -ldb -ldl -lm -lc -lcrypt -lbsd
libc=/usr/lib/libc.so.5
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef
cccdlflags='-fpic', ccdlflags='-rdynamic', lddlflags='-shared'


Alan Burlison aburlison@cix.compulink.co.uk
Re: BUG REPORT: Script that demonstrates a memory leak in 5.002beta1 [ In reply to ]
gsar at engin
Dec 9, 1995, 10:12 PM
Post #2 of 4 (406 views)
Permalink
On Sat, 09 Dec 1995 21:12:00 GMT, Alan Burlison wrote:
>The following short script leaks like a sieve on 5.002beta1, but not at
>all on 5.001m. I have included some memory statistics, and the output of
>myconfig. Some curious things: The number of items pushed onto the
>anonymous array referenced by $leak is critical - any less than 125 and
>the program size remains constant. If $leak is changed to @leak, the
>problem also disappears. Could someone a) confirm that there is a
>problem and b) if so, shed some light on what is causing it?

Here's a summary of the problem and a workaround.

The new sv_add_arena() functionality is used to reclaim AvARRAYs and
HvARRAYs that have a significantly large size. This memory gets added as a
fresh new sv arena. The problem arises when arrays and hashes grow beyond
these significant sizes inside loops which triggers relocation to a larger
contiguous block of memory and reclamation of the old one. Since the
reclamation code is not cognizant of whether it is sitting inside a loop,
this amounts to unconditional scavenging (but not free()ing) of memory.

The immediate workaround is to either compile both hv.c and av.c with
-DSTRANGE_MALLOC, or apply the following slightly more efficient patch (a
bunch of ifdefs really) that always frees [HA]vARRAYs. Until we figure out
a way to put a cap on the amount of reclaimed (but still unused) space and
make sv_add_arena() be conditional based on that, this is may be the
solution.

Doing this also eliminates all symptoms of the bug reported by Andreas a few
days back.

For reference, here are the two test cases that work fine in 5.001n:

while (1) { $leak = {}; for (1..68) { $leak->{$_} = "z"; } }
while (1) { $leak = []; for (1..125) { push(@$leak, "Z"); } }

- Sarathy.
gsar@engin.umich.edu

P.S: A couple of notes:

* I don't know why Larry did not do a sv_add_arena() in av_undef()
but did so in hv_undef()
* why AvMAX(av) rather than AvMAX(av)+1 in the sv_add_arena() call in
av_extend() ?
------------------------------------8<-----------------------------
*** av.c.dist Wed Nov 15 14:26:17 1995
--- av.c Sat Dec 9 23:03:32 1995
***************
*** 82,90 ****
newmax = tmp - 1;
New(2,ary, newmax+1, SV*);
Copy(AvALLOC(av), ary, AvMAX(av)+1, SV*);
if (AvMAX(av) > 64 && !AvREUSED(av))
! sv_add_arena((char*)AvALLOC(av), AvMAX(av) * sizeof(SV*),0);
else
Safefree(AvALLOC(av));
AvALLOC(av) = ary;
#endif
--- 82,92 ----
newmax = tmp - 1;
New(2,ary, newmax+1, SV*);
Copy(AvALLOC(av), ary, AvMAX(av)+1, SV*);
+ #ifdef 0
if (AvMAX(av) > 64 && !AvREUSED(av))
! sv_add_arena((char*)AvALLOC(av), (AvMAX(av)+1) * sizeof(SV*),0);
else
+ #endif
Safefree(AvALLOC(av));
AvALLOC(av) = ary;
#endif
*** hv.c.dist Wed Nov 15 15:00:05 1995
--- hv.c Sat Dec 9 23:10:03 1995
***************
*** 337,342 ****
--- 337,343 ----
assert(tmp >= newsize);
New(2,a, tmp, HE*);
Copy(xhv->xhv_array, a, oldsize, HE*);
+ #if 0
if (oldsize >= 64 && *(char*)&xhv->xnv_nv == 0) {
sv_add_arena((char*)xhv->xhv_array, oldsize * sizeof(HE*), 0);
sv_add_arena(((char*)a) + newsize * sizeof(HE*),
***************
*** 344,349 ****
--- 345,351 ----
SVf_FAKE);
}
else
+ #endif
Safefree(xhv->xhv_array);
#endif

***************
*** 478,487 ****
--- 480,493 ----
#ifdef STRANGE_MALLOC
Safefree(xhv->xhv_array);
#else
+ #if 0
if (xhv->xhv_max < 127 || *(char*)&xhv->xnv_nv)
Safefree(xhv->xhv_array);
else /* We used last half, so use first half for SV arena too. */
sv_add_arena((char*)xhv->xhv_array, (xhv->xhv_max + 1) * sizeof(HE*),0);
+ #else
+ Safefree(xhv->xhv_array);
+ #endif
#endif
if (HvNAME(hv)) {
Safefree(HvNAME(hv));
Re: BUG REPORT: Script that demonstrates a memory leak in 5.002beta1 [ In reply to ]
gsar at engin
Dec 9, 1995, 10:12 PM
Post #3 of 4 (408 views)
Permalink
On Sat, 09 Dec 1995 21:12:00 GMT, Alan Burlison wrote:
>The following short script leaks like a sieve on 5.002beta1, but not at
>all on 5.001m. I have included some memory statistics, and the output of
>myconfig. Some curious things: The number of items pushed onto the
>anonymous array referenced by $leak is critical - any less than 125 and
>the program size remains constant. If $leak is changed to @leak, the
>problem also disappears. Could someone a) confirm that there is a
>problem and b) if so, shed some light on what is causing it?

Here's a summary of the problem and a workaround.

The new sv_add_arena() functionality is used to reclaim AvARRAYs and
HvARRAYs that have a significantly large size. This memory gets added as a
fresh new sv arena. The problem arises when arrays and hashes grow beyond
these significant sizes inside loops which triggers relocation to a larger
contiguous block of memory and reclamation of the old one. Since the
reclamation code is not cognizant of whether it is sitting inside a loop,
this amounts to unconditional scavenging (but not free()ing) of memory.

The immediate workaround is to either compile both hv.c and av.c with
-DSTRANGE_MALLOC, or apply the following slightly more efficient patch (a
bunch of ifdefs really) that always frees [HA]vARRAYs. Until we figure out
a way to put a cap on the amount of reclaimed (but still unused) space and
make sv_add_arena() be conditional based on that, this is may be the
solution.

Doing this also eliminates all symptoms of the bug reported by Andreas a few
days back.

For reference, here are the two test cases that work fine in 5.001n:

while (1) { $leak = {}; for (1..68) { $leak->{$_} = "z"; } }
while (1) { $leak = []; for (1..125) { push(@$leak, "Z"); } }

- Sarathy.
gsar@engin.umich.edu

P.S: A couple of notes:

* I don't know why Larry did not do a sv_add_arena() in av_undef()
but did so in hv_undef()
* why AvMAX(av) rather than AvMAX(av)+1 in the sv_add_arena() call in
av_extend() ?
------------------------------------8<-----------------------------
*** av.c.dist Wed Nov 15 14:26:17 1995
--- av.c Sat Dec 9 23:03:32 1995
***************
*** 82,90 ****
newmax = tmp - 1;
New(2,ary, newmax+1, SV*);
Copy(AvALLOC(av), ary, AvMAX(av)+1, SV*);
if (AvMAX(av) > 64 && !AvREUSED(av))
! sv_add_arena((char*)AvALLOC(av), AvMAX(av) * sizeof(SV*),0);
else
Safefree(AvALLOC(av));
AvALLOC(av) = ary;
#endif
--- 82,92 ----
newmax = tmp - 1;
New(2,ary, newmax+1, SV*);
Copy(AvALLOC(av), ary, AvMAX(av)+1, SV*);
+ #ifdef 0
if (AvMAX(av) > 64 && !AvREUSED(av))
! sv_add_arena((char*)AvALLOC(av), (AvMAX(av)+1) * sizeof(SV*),0);
else
+ #endif
Safefree(AvALLOC(av));
AvALLOC(av) = ary;
#endif
*** hv.c.dist Wed Nov 15 15:00:05 1995
--- hv.c Sat Dec 9 23:10:03 1995
***************
*** 337,342 ****
--- 337,343 ----
assert(tmp >= newsize);
New(2,a, tmp, HE*);
Copy(xhv->xhv_array, a, oldsize, HE*);
+ #if 0
if (oldsize >= 64 && *(char*)&xhv->xnv_nv == 0) {
sv_add_arena((char*)xhv->xhv_array, oldsize * sizeof(HE*), 0);
sv_add_arena(((char*)a) + newsize * sizeof(HE*),
***************
*** 344,349 ****
--- 345,351 ----
SVf_FAKE);
}
else
+ #endif
Safefree(xhv->xhv_array);
#endif

***************
*** 478,487 ****
--- 480,493 ----
#ifdef STRANGE_MALLOC
Safefree(xhv->xhv_array);
#else
+ #if 0
if (xhv->xhv_max < 127 || *(char*)&xhv->xnv_nv)
Safefree(xhv->xhv_array);
else /* We used last half, so use first half for SV arena too. */
sv_add_arena((char*)xhv->xhv_array, (xhv->xhv_max + 1) * sizeof(HE*),0);
+ #else
+ Safefree(xhv->xhv_array);
+ #endif
#endif
if (HvNAME(hv)) {
Safefree(HvNAME(hv));
Re: BUG REPORT: Script that demonstrates a memory leak in 5.002beta1 [ In reply to ]
aburlison at cix
Dec 10, 1995, 3:01 PM
Post #4 of 4 (409 views)
Permalink
In-Reply-To: <199512100512.AAA11499@aatma.engin.umich.edu>
Sarathy,

> The immediate workaround is to either compile both hv.c and av.c with
> -DSTRANGE_MALLOC, or apply the following slightly more efficient patch

Many thanks for the two patches. I've applied them both and can confirm
that the problem has now disappeared - well done! One of these days I
might even get enough time to start understanding the Perl internals...
One comment, although the patches mean that my application no longer
leaks, it does seem to be slightly *less* memory efficient than 5.001m.
5.001m required about 750K of memory, the same script under 5.002beta1
needs about 840K. I'll be away in Paris next week, hopefully the 5.002
release will be out when I get back ;-)

Alan Burlison aburlison@cix.compulink.co.uk

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal