Mailing List Archive

consoles and memory leaks (was Re: How many consoles?)
On Fri, Jun 07, 2002 at 08:53:31AM -0700, David K. Z. Harris wrote:
> (Just remember the math in your configuration file (cons.h) before
> you do the build...we're using maxgrp=96, but just make sure that
> you can spawn enough processes to cover all of the ports that you
> want to connect to, and make sure your host has enough RAM for
> all of the processes.)

if you're using pre-7.2.X code, this is true. hopefully everyone's
upgraded to 7.2.2, which ignores the concept of maxgrp and just forks
processes while it can. if you have more processes than you care for,
you can have each process manage more consoles by using the -m flag
(default is 16).

On Fri, Jun 07, 2002 at 12:02:29PM -0400, Greg A. Woods wrote:
> That said, as you can see above there's still a very serious memory
> leak somewhere, and that'll drastically affect scalability in any
> production system.... The second conserver process is allocating an
> additional 1.2KB or so with every client connection (I have a cron job
> that makes console client connections to collect data from my three UPS
> units once every minute).

hmmm...i just did a test with 7.2.2 (and i don't know why it would be
different with 7.2.1) by having conserver manage five consoles. i then
did:

i=1; while true; do (echo "echo $1"; sleep 1) | \
~/./conserver/playpen/conserver-7.2.2/console/console bash; \
i=$((i + 1)); done

it's currently spewing:

underdog (root) 288:# [Enter `^Ec?' for help]
echo 287
287
underdog (root) 289:#

and still going. the memory size hasn't grown at all (using both pmap
and ps to check sizes - solaris here). i also have a redhat 7.3 system
running the same test - just started it up. it's up to 109 connections
and there hasn't been a change in memory usage, according to ps.

well, we're up to 983 on the solaris test - still no change in memory
usage. i don't know if it's a netbsd library leak, or maybe the
solaris and linux libraries are more forgiving about something, but i
can't reproduce the problem. if you or anyone can pin things down, i'd
love to hear about it. at 307 and 1130 now - still no change.

if, for some reason, the clients aren't getting disconnected, you'd
have a constant memory increase. but, you should either have a whole
lot of console binaries running, or if you do a 'console -w', you'd see
a list of clients on the console. i kinda doubt that's what's going
on, but i wanted to mention it.

Bryan
Re: consoles, client, and memory leaks [ In reply to ]
[. On Friday, June 7, 2002 at 10:40:25 (-0700), Bryan Stansell wrote: ]
> Subject: consoles and memory leaks (was Re: How many consoles?)
>
> On Fri, Jun 07, 2002 at 12:02:29PM -0400, Greg A. Woods wrote:
> > That said, as you can see above there's still a very serious memory
> > leak somewhere, and that'll drastically affect scalability in any
> > production system.... The second conserver process is allocating an
> > additional 1.2KB or so with every client connection (I have a cron job
> > that makes console client connections to collect data from my three UPS
> > units once every minute).
>
> hmmm...i just did a test with 7.2.2 (and i don't know why it would be
> different with 7.2.1) by having conserver manage five consoles. i then
> did:
>
> i=1; while true; do (echo "echo $1"; sleep 1) | \
> ~/./conserver/playpen/conserver-7.2.2/console/console bash; \
> i=$((i + 1)); done
>
> it's currently spewing:
>
> underdog (root) 288:# [Enter `^Ec?' for help]
> echo 287
> 287
> underdog (root) 289:#
>
> and still going. the memory size hasn't grown at all (using both pmap
> and ps to check sizes - solaris here). i also have a redhat 7.3 system
> running the same test - just started it up. it's up to 109 connections
> and there hasn't been a change in memory usage, according to ps.

Well I've finally upgraded to 7.2.2 and re-integrated my 'chat' patches,
but I'm still seeing the memory leak on my good old NetBSD-1.3.2 server.

After re-jigging the debug output in many places, and keeping much more
careful track of when file descriptors are created and closed, I've
finally narrowed it down to somewhere in the massive Kiddie() function
(or something it calls) in the "MAIN" while(1) loop. It looks like its
closer to an ~2KB increase for every client connection. I can't see any
obvious problems yet though. it looks like all the file descriptors are
getting closed properly and so on. All the new chat code is quiescent
during this time too.

I'm going to link against libefence and/or libvmalloc and see if I can't
pin it down....

--
Greg A. Woods

+1 416 218-0098; <g.a.woods@ieee.org>; <woods@robohack.ca>
Planix, Inc. <woods@planix.com>; VE3TCP; Secrets of the Weird <woods@weird.com>
Re: consoles, client, and memory leaks [ In reply to ]
I'm slaming my server with conserver connections to test this problem.
I'm not seeing anything but I am using a heavily modified version of
7.2.2. I'm doing my test on Linux 2.4.18.

Maybe something in the NetBSD build. Make sure all malloc()'s have a
free().


On Sun, 2002-09-22 at 12:26, Greg A. Woods wrote:
> [. On Friday, June 7, 2002 at 10:40:25 (-0700), Bryan Stansell wrote: ]
> > Subject: consoles and memory leaks (was Re: How many consoles?)
> >
> > On Fri, Jun 07, 2002 at 12:02:29PM -0400, Greg A. Woods wrote:
> > > That said, as you can see above there's still a very serious memory
> > > leak somewhere, and that'll drastically affect scalability in any
> > > production system.... The second conserver process is allocating an
> > > additional 1.2KB or so with every client connection (I have a cron job
> > > that makes console client connections to collect data from my three UPS
> > > units once every minute).
> >
> > hmmm...i just did a test with 7.2.2 (and i don't know why it would be
> > different with 7.2.1) by having conserver manage five consoles. i then
> > did:
> >
> > i=1; while true; do (echo "echo $1"; sleep 1) | \
> > ~/./conserver/playpen/conserver-7.2.2/console/console bash; \
> > i=$((i + 1)); done
> >
> > it's currently spewing:
> >
> > underdog (root) 288:# [Enter `^Ec?' for help]
> > echo 287
> > 287
> > underdog (root) 289:#
> >
> > and still going. the memory size hasn't grown at all (using both pmap
> > and ps to check sizes - solaris here). i also have a redhat 7.3 system
> > running the same test - just started it up. it's up to 109 connections
> > and there hasn't been a change in memory usage, according to ps.
>
> Well I've finally upgraded to 7.2.2 and re-integrated my 'chat' patches,
> but I'm still seeing the memory leak on my good old NetBSD-1.3.2 server.
>
> After re-jigging the debug output in many places, and keeping much more
> careful track of when file descriptors are created and closed, I've
> finally narrowed it down to somewhere in the massive Kiddie() function
> (or something it calls) in the "MAIN" while(1) loop. It looks like its
> closer to an ~2KB increase for every client connection. I can't see any
> obvious problems yet though. it looks like all the file descriptors are
> getting closed properly and so on. All the new chat code is quiescent
> during this time too.
>
> I'm going to link against libefence and/or libvmalloc and see if I can't
> pin it down....
>
> --
> Greg A. Woods
>
> +1 416 218-0098; <g.a.woods@ieee.org>; <woods@robohack.ca>
> Planix, Inc. <woods@planix.com>; VE3TCP; Secrets of the Weird <woods@weird.com>
> _______________________________________________
> users mailing list
> users@conserver.com
> https://www.conserver.com/mailman/listinfo/users
>
>
>