Mailing List Archive

rsyslog threads questions
Hi,

I have 3.16.2 which was recently released. I see that under certain
conditions rsyslogd spawns a lot of threads:
5949 root 11216 S rsyslogd
5950 root 11216 S rsyslogd
5951 root 11216 S rsyslogd
5952 root 11216 S rsyslogd
5953 root 11216 S rsyslogd
5954 root 11216 S rsyslogd
5985 root Z [rsyslogd]
6445 root Z [rsyslogd]

I had to kill the rsyslogd and restart it. The first invocation had a
pid of 219 before it had to be killed. The second invocation of pid
which you see above starts with 5949. The difference is the amount of
zombie threads that were invoked by rsyslogd before I had to kill the
first invocation of it.

The question is under what conditions does rsyslogd spawn a new
thread/process and why was it a zombie? I am running rsyslogd in an
embedded environment and not a regular laptop/desktop. In addition, I
am using busybox and I believe the syslog buffer size is set to
something very low or perhaps none at all. Would this be a factor?
Furthermore, I ran rsyslogd with -c3 and also without -c3 and both
cases happen.

Are these issues already known and fixed in a later version? Sorry, if
I am asking the same questions or have the same issues as previous
people but without the ability to search (or at least, I don't know
how to) the archive, I don't know if my problem/questions has already
been seen and/or resolved.

Thank you very much for your support.

Scott
rsyslog threads questions [ In reply to ]
On Wed, 2008-07-02 at 01:15 -0700, Scott Phuong wrote:
> Hi,
>
> I have 3.16.2 which was recently released. I see that under certain
> conditions rsyslogd spawns a lot of threads:
> 5949 root 11216 S rsyslogd
> 5950 root 11216 S rsyslogd
> 5951 root 11216 S rsyslogd
> 5952 root 11216 S rsyslogd
> 5953 root 11216 S rsyslogd
> 5954 root 11216 S rsyslogd
> 5985 root Z [rsyslogd]
> 6445 root Z [rsyslogd]
>
> I had to kill the rsyslogd and restart it. The first invocation had a
> pid of 219 before it had to be killed. The second invocation of pid
> which you see above starts with 5949. The difference is the amount of
> zombie threads that were invoked by rsyslogd before I had to kill the
> first invocation of it.

I have no explanation yet for the zombies. They should not happen and so
far I have never seen them. We may need to go through a debug log (which
will become very large) to find out what's going on.

> The question is under what conditions does rsyslogd spawn a new
> thread/process and why was it a zombie?

Unfortunately, there is no quick answer. A quick one may be: when it
needs them, based on queue watermark settings and based on you
configuration. But to really understand it, you need to read this doc:

http://www.rsyslog.com/doc-queues.html

The doc also describes all the knobs that you can use to control thread
creation. There are many ;)

> I am running rsyslogd in an
> embedded environment and not a regular laptop/desktop.

Interesting use case...

> In addition, I
> am using busybox and I believe the syslog buffer size is set to

what do yo mean by "syslog buffer size"? The length of a receive buffer?
It is 2K, thus single messages up to 2K are supported. It can be changed
by modifying the MAXLINE define. Note that stock syslogd (and RFC3164)
support only up to 1K.

> something very low or perhaps none at all. Would this be a factor?
> Furthermore, I ran rsyslogd with -c3 and also without -c3 and both
> cases happen.

The compatibility modes do not affect queue operation.

> Are these issues already known and fixed in a later version? Sorry, if
> I am asking the same questions or have the same issues as previous
> people but without the ability to search (or at least, I don't know
> how to) the archive, I don't know if my problem/questions has already
> been seen and/or resolved.

If we need to find out about the zombies, we need to move on to the
current devel version. So I would give that a try in any case. 3.16.2
will (most probably) be replaced by 3.18.0 (based on the current beta)
next week. So I won't touch it any longer.

Looking forward to your feedback,
Rainer

>
> Thank you very much for your support.
>
> Scott
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
rsyslog threads questions [ In reply to ]
Hi Rainer,

Thanks for your reply. Looking at the default settings (from the
online help's configuration page), they are what I wanted. The main
messages queue is set to fix sized array with 1 worker thread created
at maximum and action queues are direct mode which according to the
queue document page, means that there will not be a worker thread
created. Is my understanding correct? If yes, how do I quickly check
without using the -d option if the defaults are set correctly? Or what
do I look for in the debug messages that gets printed out to ensure
this?

You also mentioned that version 3.18.0 is probably going to be
released as the stable version next week. I see on the webpage there
is a 3.17.4 and 3.17.5. Are these two versions similiar to 3.18.0?

Also, how come I did not get your reply in my email inbox? My account
settings look correct.

Thanks,

Scott Phuong

As for the syslog buffer size, that applies to syslogd and does not
apply to rsyslog.



My configuration files do not change the Action queue or Worker queue
parameters at all. Looking at
On Wed, 2008-07-02 at 01:15 -0700, Scott Phuong wrote:
> Hi,
>
> I have 3.16.2 which was recently released. I see that under certain
> conditions rsyslogd spawns a lot of threads:
> 5949 root 11216 S rsyslogd
> 5950 root 11216 S rsyslogd
> 5951 root 11216 S rsyslogd
> 5952 root 11216 S rsyslogd
> 5953 root 11216 S rsyslogd
> 5954 root 11216 S rsyslogd
> 5985 root Z [rsyslogd]
> 6445 root Z [rsyslogd]
>
> I had to kill the rsyslogd and restart it. The first invocation had a
> pid of 219 before it had to be killed. The second invocation of pid
> which you see above starts with 5949. The difference is the amount of
> zombie threads that were invoked by rsyslogd before I had to kill the
> first invocation of it.

I have no explanation yet for the zombies. They should not happen and so
far I have never seen them. We may need to go through a debug log (which
will become very large) to find out what's going on.

> The question is under what conditions does rsyslogd spawn a new
> thread/process and why was it a zombie?

Unfortunately, there is no quick answer. A quick one may be: when it
needs them, based on queue watermark settings and based on you
configuration. But to really understand it, you need to read this doc:

http://www.rsyslog.com/doc-queues.html

The doc also describes all the knobs that you can use to control thread
creation. There are many ;)

> I am running rsyslogd in an
> embedded environment and not a regular laptop/desktop.

Interesting use case...

> In addition, I
> am using busybox and I believe the syslog buffer size is set to

what do yo mean by "syslog buffer size"? The length of a receive buffer?
It is 2K, thus single messages up to 2K are supported. It can be changed
by modifying the MAXLINE define. Note that stock syslogd (and RFC3164)
support only up to 1K.

> something very low or perhaps none at all. Would this be a factor?
> Furthermore, I ran rsyslogd with -c3 and also without -c3 and both
> cases happen.

The compatibility modes do not affect queue operation.

> Are these issues already known and fixed in a later version? Sorry, if
> I am asking the same questions or have the same issues as previous
> people but without the ability to search (or at least, I don't know
> how to) the archive, I don't know if my problem/questions has already
> been seen and/or resolved.

If we need to find out about the zombies, we need to move on to the
current devel version. So I would give that a try in any case. 3.16.2
will (most probably) be replaced by 3.18.0 (based on the current beta)
next week. So I won't touch it any longer.

Looking forward to your feedback,
Rainer

>
> Thank you very much for your support.
>
> Scott
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
rsyslog threads questions [ In reply to ]
Hi,

I've attached four files. Two of which are debug dumps, one is the
conf file and the last one is a test case scenario that constantly
fails on my end. I hope this gives a little more information.
Furthermore, the dumps are from 3.17.5 which is the "closest" version
to 3.18.0 that I was able to find.

Both failed scenarios occur when lots of messages were being flooded
to rsyslogd at a very fast rate (look at logtest.c) The
my_arm_rsyslog_suicide_debug.txt received a sigsegv fault while
my_arm_rsyslog_sh_cannot_fork caused so many "Z [rsyslogd]" threads
that it took up so much memory that executing any command as simple as
'ls -l' would not work from the command line. I think the number of
threads grew as much as the number of messages. In the latter
scenario, after killing logtest.c, it didn't look like the those
zombies threads went away until I did a CTRL+C to the rsyslogd which
was running in the foreground since I use the "-dn" option.

This is on an embedded system that runs significantly slower than a
desktop or laptop so maybe it would be harder to reproduce on a
regular computer. I looked at all the parameters that I believe could
affect this and believe for the most part the defaults are more than
adequate. The main message queue never looked like it hit the high
water mark but it did hit the lower one. So, I don't think messages
were being dropped (not sure) or an overflow condition occurred.

The processor is ARM-based and it is using Linux kernel 2.6.16.12 and
compiled using GCC and the standard GNU C libraries version 3.4.5.
Rsyslog source code is cross-compiled using the following configure
line:

./configure --disable-zlib --disable-largefile
--enable-share=yes
--prefix=/
--host=arm-unknown-linux-gnu
ac_cv_func_malloc_0_nonnull=yes
ac_cv_func_realloc_0_nonnull=yes
ac_cv_func_lstat_dereferences_slashed_symlink=yes
ac_cv_func_stat_empty_string_bug=no
enable_debug=no
enable_rtinst=no

Lastly, the logtest was executed with just the "-s" parameter. It is a
simple C file that I came up with.

I took a look at the debug messages and it does not appear that new
threads are created via calls to wtpStartWrkr in wtp.c.

Any help I can bring to solve this issue, please let me know. I hope I
am not doing anything wrong here.

Thanks,

Scott
>
> On Wed, Jul 2, 2008 at 12:04 PM, Scott Phuong <mycleanjunk at gmail.com> wrote:
>> Hi Rainer,
>>
>> Thanks for your reply. Looking at the default settings (from the
>> online help's configuration page), they are what I wanted. The main
>> messages queue is set to fix sized array with 1 worker thread created
>> at maximum and action queues are direct mode which according to the
>> queue document page, means that there will not be a worker thread
>> created. Is my understanding correct? If yes, how do I quickly check
>> without using the -d option if the defaults are set correctly? Or what
>> do I look for in the debug messages that gets printed out to ensure
>> this?
>>
>> You also mentioned that version 3.18.0 is probably going to be
>> released as the stable version next week. I see on the webpage there
>> is a 3.17.4 and 3.17.5. Are these two versions similiar to 3.18.0?
>>
>> Also, how come I did not get your reply in my email inbox? My account
>> settings look correct.
>>
>> Thanks,
>>
>> Scott Phuong
>>
>> As for the syslog buffer size, that applies to syslogd and does not
>> apply to rsyslog.
>>
>>
>>
>> My configuration files do not change the Action queue or Worker queue
>> parameters at all. Looking at
>> On Wed, 2008-07-02 at 01:15 -0700, Scott Phuong wrote:
>>> Hi,
>>>
>>> I have 3.16.2 which was recently released. I see that under certain
>>> conditions rsyslogd spawns a lot of threads:
>>> 5949 root 11216 S rsyslogd
>>> 5950 root 11216 S rsyslogd
>>> 5951 root 11216 S rsyslogd
>>> 5952 root 11216 S rsyslogd
>>> 5953 root 11216 S rsyslogd
>>> 5954 root 11216 S rsyslogd
>>> 5985 root Z [rsyslogd]
>>> 6445 root Z [rsyslogd]
>>>
>>> I had to kill the rsyslogd and restart it. The first invocation had a
>>> pid of 219 before it had to be killed. The second invocation of pid
>>> which you see above starts with 5949. The difference is the amount of
>>> zombie threads that were invoked by rsyslogd before I had to kill the
>>> first invocation of it.
>>
>> I have no explanation yet for the zombies. They should not happen and so
>> far I have never seen them. We may need to go through a debug log (which
>> will become very large) to find out what's going on.
>>
>>> The question is under what conditions does rsyslogd spawn a new
>>> thread/process and why was it a zombie?
>>
>> Unfortunately, there is no quick answer. A quick one may be: when it
>> needs them, based on queue watermark settings and based on you
>> configuration. But to really understand it, you need to read this doc:
>>
>> http://www.rsyslog.com/doc-queues.html
>>
>> The doc also describes all the knobs that you can use to control thread
>> creation. There are many ;)
>>
>>> I am running rsyslogd in an
>>> embedded environment and not a regular laptop/desktop.
>>
>> Interesting use case...
>>
>>> In addition, I
>>> am using busybox and I believe the syslog buffer size is set to
>>
>> what do yo mean by "syslog buffer size"? The length of a receive buffer?
>> It is 2K, thus single messages up to 2K are supported. It can be changed
>> by modifying the MAXLINE define. Note that stock syslogd (and RFC3164)
>> support only up to 1K.
>>
>>> something very low or perhaps none at all. Would this be a factor?
>>> Furthermore, I ran rsyslogd with -c3 and also without -c3 and both
>>> cases happen.
>>
>> The compatibility modes do not affect queue operation.
>>
>>> Are these issues already known and fixed in a later version? Sorry, if
>>> I am asking the same questions or have the same issues as previous
>>> people but without the ability to search (or at least, I don't know
>>> how to) the archive, I don't know if my problem/questions has already
>>> been seen and/or resolved.
>>
>> If we need to find out about the zombies, we need to move on to the
>> current devel version. So I would give that a try in any case. 3.16.2
>> will (most probably) be replaced by 3.18.0 (based on the current beta)
>> next week. So I won't touch it any longer.
>>
>> Looking forward to your feedback,
>> Rainer
>>
>>>
>>> Thank you very much for your support.
>>>
>>> Scott
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>
>
rsyslog threads questions [ In reply to ]
Sorry for sending a lot of emails. I believe I have root-cause the
issue in 3.17.5. It appears that when the rsyslog.conf file does
something like this:
emerg.* *;mytemplate

The code executes wallmsg which does forks. This is how I got a lot of
processes were being created in my scenario. Although one would hope
that a system doesn't misbehave this awful, nonetheless it does
happen. First, it seems that the emergency messages should go to all
those that are logged in via whatever mechanism (i.e. telnet, console,
stty, ssh, etc...), this message should appear on all those "screens".
I see that it does not even if I just logged 1 message at this
serverity. I believe this is how syslog described setting up a wall
message and what it is. Is this a bug?

Lastly, when the child of the fork exits, it does not appear to be
removed from Linux and still continues to eat up memory and is
reported as a zombie. It appears that when the workerthread has been
"destroyed/deconstructed" does things clear up. I am not sure if this
is a rsyslogd problem or a linux or gcc problem. Any ideas?

Thanks,

Scott



On Wed, Jul 2, 2008 at 5:57 PM, Scott Phuong <mycleanjunk at gmail.com> wrote:
> Hi,
>
> I've attached four files. Two of which are debug dumps, one is the
> conf file and the last one is a test case scenario that constantly
> fails on my end. I hope this gives a little more information.
> Furthermore, the dumps are from 3.17.5 which is the "closest" version
> to 3.18.0 that I was able to find.
>
> Both failed scenarios occur when lots of messages were being flooded
> to rsyslogd at a very fast rate (look at logtest.c) The
> my_arm_rsyslog_suicide_debug.txt received a sigsegv fault while
> my_arm_rsyslog_sh_cannot_fork caused so many "Z [rsyslogd]" threads
> that it took up so much memory that executing any command as simple as
> 'ls -l' would not work from the command line. I think the number of
> threads grew as much as the number of messages. In the latter
> scenario, after killing logtest.c, it didn't look like the those
> zombies threads went away until I did a CTRL+C to the rsyslogd which
> was running in the foreground since I use the "-dn" option.
>
> This is on an embedded system that runs significantly slower than a
> desktop or laptop so maybe it would be harder to reproduce on a
> regular computer. I looked at all the parameters that I believe could
> affect this and believe for the most part the defaults are more than
> adequate. The main message queue never looked like it hit the high
> water mark but it did hit the lower one. So, I don't think messages
> were being dropped (not sure) or an overflow condition occurred.
>
> The processor is ARM-based and it is using Linux kernel 2.6.16.12 and
> compiled using GCC and the standard GNU C libraries version 3.4.5.
> Rsyslog source code is cross-compiled using the following configure
> line:
>
> ./configure --disable-zlib --disable-largefile
> --enable-share=yes
> --prefix=/
> --host=arm-unknown-linux-gnu
> ac_cv_func_malloc_0_nonnull=yes
> ac_cv_func_realloc_0_nonnull=yes
> ac_cv_func_lstat_dereferences_slashed_symlink=yes
> ac_cv_func_stat_empty_string_bug=no
> enable_debug=no
> enable_rtinst=no
>
> Lastly, the logtest was executed with just the "-s" parameter. It is a
> simple C file that I came up with.
>
> I took a look at the debug messages and it does not appear that new
> threads are created via calls to wtpStartWrkr in wtp.c.
>
> Any help I can bring to solve this issue, please let me know. I hope I
> am not doing anything wrong here.
>
> Thanks,
>
> Scott
>>
>> On Wed, Jul 2, 2008 at 12:04 PM, Scott Phuong <mycleanjunk at gmail.com> wrote:
>>> Hi Rainer,
>>>
>>> Thanks for your reply. Looking at the default settings (from the
>>> online help's configuration page), they are what I wanted. The main
>>> messages queue is set to fix sized array with 1 worker thread created
>>> at maximum and action queues are direct mode which according to the
>>> queue document page, means that there will not be a worker thread
>>> created. Is my understanding correct? If yes, how do I quickly check
>>> without using the -d option if the defaults are set correctly? Or what
>>> do I look for in the debug messages that gets printed out to ensure
>>> this?
>>>
>>> You also mentioned that version 3.18.0 is probably going to be
>>> released as the stable version next week. I see on the webpage there
>>> is a 3.17.4 and 3.17.5. Are these two versions similiar to 3.18.0?
>>>
>>> Also, how come I did not get your reply in my email inbox? My account
>>> settings look correct.
>>>
>>> Thanks,
>>>
>>> Scott Phuong
>>>
>>> As for the syslog buffer size, that applies to syslogd and does not
>>> apply to rsyslog.
>>>
>>>
>>>
>>> My configuration files do not change the Action queue or Worker queue
>>> parameters at all. Looking at
>>> On Wed, 2008-07-02 at 01:15 -0700, Scott Phuong wrote:
>>>> Hi,
>>>>
>>>> I have 3.16.2 which was recently released. I see that under certain
>>>> conditions rsyslogd spawns a lot of threads:
>>>> 5949 root 11216 S rsyslogd
>>>> 5950 root 11216 S rsyslogd
>>>> 5951 root 11216 S rsyslogd
>>>> 5952 root 11216 S rsyslogd
>>>> 5953 root 11216 S rsyslogd
>>>> 5954 root 11216 S rsyslogd
>>>> 5985 root Z [rsyslogd]
>>>> 6445 root Z [rsyslogd]
>>>>
>>>> I had to kill the rsyslogd and restart it. The first invocation had a
>>>> pid of 219 before it had to be killed. The second invocation of pid
>>>> which you see above starts with 5949. The difference is the amount of
>>>> zombie threads that were invoked by rsyslogd before I had to kill the
>>>> first invocation of it.
>>>
>>> I have no explanation yet for the zombies. They should not happen and so
>>> far I have never seen them. We may need to go through a debug log (which
>>> will become very large) to find out what's going on.
>>>
>>>> The question is under what conditions does rsyslogd spawn a new
>>>> thread/process and why was it a zombie?
>>>
>>> Unfortunately, there is no quick answer. A quick one may be: when it
>>> needs them, based on queue watermark settings and based on you
>>> configuration. But to really understand it, you need to read this doc:
>>>
>>> http://www.rsyslog.com/doc-queues.html
>>>
>>> The doc also describes all the knobs that you can use to control thread
>>> creation. There are many ;)
>>>
>>>> I am running rsyslogd in an
>>>> embedded environment and not a regular laptop/desktop.
>>>
>>> Interesting use case...
>>>
>>>> In addition, I
>>>> am using busybox and I believe the syslog buffer size is set to
>>>
>>> what do yo mean by "syslog buffer size"? The length of a receive buffer?
>>> It is 2K, thus single messages up to 2K are supported. It can be changed
>>> by modifying the MAXLINE define. Note that stock syslogd (and RFC3164)
>>> support only up to 1K.
>>>
>>>> something very low or perhaps none at all. Would this be a factor?
>>>> Furthermore, I ran rsyslogd with -c3 and also without -c3 and both
>>>> cases happen.
>>>
>>> The compatibility modes do not affect queue operation.
>>>
>>>> Are these issues already known and fixed in a later version? Sorry, if
>>>> I am asking the same questions or have the same issues as previous
>>>> people but without the ability to search (or at least, I don't know
>>>> how to) the archive, I don't know if my problem/questions has already
>>>> been seen and/or resolved.
>>>
>>> If we need to find out about the zombies, we need to move on to the
>>> current devel version. So I would give that a try in any case. 3.16.2
>>> will (most probably) be replaced by 3.18.0 (based on the current beta)
>>> next week. So I won't touch it any longer.
>>>
>>> Looking forward to your feedback,
>>> Rainer
>>>
>>>>
>>>> Thank you very much for your support.
>>>>
>>>> Scott
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>
>>
>
rsyslog threads questions [ In reply to ]
Hi Scott,

will reply in more detail later today, but one thing quickly. This is
very good information. The wall code is ooooooooooold, stems back to
sysklogd and has only been slightly modified. I have to admit it is not
much on my testing radar. So I assume you found the bug. I think I'll
need to re-design it. With the queue engine, forking off should not
really be necessary.

I need to see if that goes into the v3-stable soon or is kept at the
devel tree. I am a bit hesitant to move it to stable, as it involves a
good chance for additional bugs...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Scott Phuong
> Sent: Thursday, July 03, 2008 8:48 AM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] rsyslog threads questions
>
> Sorry for sending a lot of emails. I believe I have root-cause the
> issue in 3.17.5. It appears that when the rsyslog.conf file does
> something like this:
> emerg.* *;mytemplate
>
> The code executes wallmsg which does forks. This is how I got a lot of
> processes were being created in my scenario. Although one would hope
> that a system doesn't misbehave this awful, nonetheless it does
> happen. First, it seems that the emergency messages should go to all
> those that are logged in via whatever mechanism (i.e. telnet, console,
> stty, ssh, etc...), this message should appear on all those "screens".
> I see that it does not even if I just logged 1 message at this
> serverity. I believe this is how syslog described setting up a wall
> message and what it is. Is this a bug?
>
> Lastly, when the child of the fork exits, it does not appear to be
> removed from Linux and still continues to eat up memory and is
> reported as a zombie. It appears that when the workerthread has been
> "destroyed/deconstructed" does things clear up. I am not sure if this
> is a rsyslogd problem or a linux or gcc problem. Any ideas?
>
> Thanks,
>
> Scott
>
>
>
> On Wed, Jul 2, 2008 at 5:57 PM, Scott Phuong <mycleanjunk at gmail.com>
> wrote:
> > Hi,
> >
> > I've attached four files. Two of which are debug dumps, one is the
> > conf file and the last one is a test case scenario that constantly
> > fails on my end. I hope this gives a little more information.
> > Furthermore, the dumps are from 3.17.5 which is the "closest"
version
> > to 3.18.0 that I was able to find.
> >
> > Both failed scenarios occur when lots of messages were being flooded
> > to rsyslogd at a very fast rate (look at logtest.c) The
> > my_arm_rsyslog_suicide_debug.txt received a sigsegv fault while
> > my_arm_rsyslog_sh_cannot_fork caused so many "Z [rsyslogd]"
> threads
> > that it took up so much memory that executing any command as simple
> as
> > 'ls -l' would not work from the command line. I think the number of
> > threads grew as much as the number of messages. In the latter
> > scenario, after killing logtest.c, it didn't look like the those
> > zombies threads went away until I did a CTRL+C to the rsyslogd which
> > was running in the foreground since I use the "-dn" option.
> >
> > This is on an embedded system that runs significantly slower than a
> > desktop or laptop so maybe it would be harder to reproduce on a
> > regular computer. I looked at all the parameters that I believe
could
> > affect this and believe for the most part the defaults are more than
> > adequate. The main message queue never looked like it hit the high
> > water mark but it did hit the lower one. So, I don't think messages
> > were being dropped (not sure) or an overflow condition occurred.
> >
> > The processor is ARM-based and it is using Linux kernel 2.6.16.12
and
> > compiled using GCC and the standard GNU C libraries version 3.4.5.
> > Rsyslog source code is cross-compiled using the following configure
> > line:
> >
> > ./configure --disable-zlib --disable-largefile
> > --enable-share=yes
> > --prefix=/
> > --host=arm-unknown-linux-gnu
> > ac_cv_func_malloc_0_nonnull=yes
> > ac_cv_func_realloc_0_nonnull=yes
> >
> ac_cv_func_lstat_dereferences_slashed_symlink=yes
> > ac_cv_func_stat_empty_string_bug=no
> > enable_debug=no
> > enable_rtinst=no
> >
> > Lastly, the logtest was executed with just the "-s" parameter. It is
> a
> > simple C file that I came up with.
> >
> > I took a look at the debug messages and it does not appear that new
> > threads are created via calls to wtpStartWrkr in wtp.c.
> >
> > Any help I can bring to solve this issue, please let me know. I hope
> I
> > am not doing anything wrong here.
> >
> > Thanks,
> >
> > Scott
> >>
> >> On Wed, Jul 2, 2008 at 12:04 PM, Scott Phuong
> <mycleanjunk at gmail.com> wrote:
> >>> Hi Rainer,
> >>>
> >>> Thanks for your reply. Looking at the default settings (from the
> >>> online help's configuration page), they are what I wanted. The
main
> >>> messages queue is set to fix sized array with 1 worker thread
> created
> >>> at maximum and action queues are direct mode which according to
the
> >>> queue document page, means that there will not be a worker thread
> >>> created. Is my understanding correct? If yes, how do I quickly
> check
> >>> without using the -d option if the defaults are set correctly? Or
> what
> >>> do I look for in the debug messages that gets printed out to
ensure
> >>> this?
> >>>
> >>> You also mentioned that version 3.18.0 is probably going to be
> >>> released as the stable version next week. I see on the webpage
> there
> >>> is a 3.17.4 and 3.17.5. Are these two versions similiar to 3.18.0?
> >>>
> >>> Also, how come I did not get your reply in my email inbox? My
> account
> >>> settings look correct.
> >>>
> >>> Thanks,
> >>>
> >>> Scott Phuong
> >>>
> >>> As for the syslog buffer size, that applies to syslogd and does
not
> >>> apply to rsyslog.
> >>>
> >>>
> >>>
> >>> My configuration files do not change the Action queue or Worker
> queue
> >>> parameters at all. Looking at
> >>> On Wed, 2008-07-02 at 01:15 -0700, Scott Phuong wrote:
> >>>> Hi,
> >>>>
> >>>> I have 3.16.2 which was recently released. I see that under
> certain
> >>>> conditions rsyslogd spawns a lot of threads:
> >>>> 5949 root 11216 S rsyslogd
> >>>> 5950 root 11216 S rsyslogd
> >>>> 5951 root 11216 S rsyslogd
> >>>> 5952 root 11216 S rsyslogd
> >>>> 5953 root 11216 S rsyslogd
> >>>> 5954 root 11216 S rsyslogd
> >>>> 5985 root Z [rsyslogd]
> >>>> 6445 root Z [rsyslogd]
> >>>>
> >>>> I had to kill the rsyslogd and restart it. The first invocation
> had a
> >>>> pid of 219 before it had to be killed. The second invocation of
> pid
> >>>> which you see above starts with 5949. The difference is the
amount
> of
> >>>> zombie threads that were invoked by rsyslogd before I had to kill
> the
> >>>> first invocation of it.
> >>>
> >>> I have no explanation yet for the zombies. They should not happen
> and so
> >>> far I have never seen them. We may need to go through a debug log
> (which
> >>> will become very large) to find out what's going on.
> >>>
> >>>> The question is under what conditions does rsyslogd spawn a new
> >>>> thread/process and why was it a zombie?
> >>>
> >>> Unfortunately, there is no quick answer. A quick one may be: when
> it
> >>> needs them, based on queue watermark settings and based on you
> >>> configuration. But to really understand it, you need to read this
> doc:
> >>>
> >>> http://www.rsyslog.com/doc-queues.html
> >>>
> >>> The doc also describes all the knobs that you can use to control
> thread
> >>> creation. There are many ;)
> >>>
> >>>> I am running rsyslogd in an
> >>>> embedded environment and not a regular laptop/desktop.
> >>>
> >>> Interesting use case...
> >>>
> >>>> In addition, I
> >>>> am using busybox and I believe the syslog buffer size is set to
> >>>
> >>> what do yo mean by "syslog buffer size"? The length of a receive
> buffer?
> >>> It is 2K, thus single messages up to 2K are supported. It can be
> changed
> >>> by modifying the MAXLINE define. Note that stock syslogd (and
> RFC3164)
> >>> support only up to 1K.
> >>>
> >>>> something very low or perhaps none at all. Would this be a
factor?
> >>>> Furthermore, I ran rsyslogd with -c3 and also without -c3 and
both
> >>>> cases happen.
> >>>
> >>> The compatibility modes do not affect queue operation.
> >>>
> >>>> Are these issues already known and fixed in a later version?
> Sorry, if
> >>>> I am asking the same questions or have the same issues as
previous
> >>>> people but without the ability to search (or at least, I don't
> know
> >>>> how to) the archive, I don't know if my problem/questions has
> already
> >>>> been seen and/or resolved.
> >>>
> >>> If we need to find out about the zombies, we need to move on to
the
> >>> current devel version. So I would give that a try in any case.
> 3.16.2
> >>> will (most probably) be replaced by 3.18.0 (based on the current
> beta)
> >>> next week. So I won't touch it any longer.
> >>>
> >>> Looking forward to your feedback,
> >>> Rainer
> >>>
> >>>>
> >>>> Thank you very much for your support.
> >>>>
> >>>> Scott
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>
> >>
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
rsyslog threads questions [ In reply to ]
Scott,

I have rewritten the omusrmsg plugin. This is currently part of the
development tree, but will become part of the (new) beta when we do a
version number shift next week. I'd appreciate if you could give it a
try. It is available here:

http://download.rsyslog.com/download/rsyslog-3.19.9-pre1.tar.gz

I am hesitant to put this in the upcoming new v3-stable. There were lots
of changes and one rule for a stable is that there should be no non-fix
type of changes for at least 4 weeks before it turns into a stable. It
of course is debatable if the change I made is a fix or not. In some
sense it is, but on the other hand it is a rewrite that changed a lot.
So given the fact that nobody on a "regular" machine saw an issue the
past year, I would not like to bring the new code directly into the
stable. If that causes problems for you, you may want to try simply
using the omusrmsg.c code together with the v3-stable. I haven't tried
it, but it should work well. All changes were just to that file, and
there are no dependencies to anything external.

OK... but now let's see first if it fixes the issue ;)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Scott Phuong
> Sent: Thursday, July 03, 2008 2:58 AM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] rsyslog threads questions
>
> Hi,
>
> I've attached four files. Two of which are debug dumps, one is the
> conf file and the last one is a test case scenario that constantly
> fails on my end. I hope this gives a little more information.
> Furthermore, the dumps are from 3.17.5 which is the "closest" version
> to 3.18.0 that I was able to find.
>
> Both failed scenarios occur when lots of messages were being flooded
> to rsyslogd at a very fast rate (look at logtest.c) The
> my_arm_rsyslog_suicide_debug.txt received a sigsegv fault while
> my_arm_rsyslog_sh_cannot_fork caused so many "Z [rsyslogd]" threads
> that it took up so much memory that executing any command as simple as
> 'ls -l' would not work from the command line. I think the number of
> threads grew as much as the number of messages. In the latter
> scenario, after killing logtest.c, it didn't look like the those
> zombies threads went away until I did a CTRL+C to the rsyslogd which
> was running in the foreground since I use the "-dn" option.
>
> This is on an embedded system that runs significantly slower than a
> desktop or laptop so maybe it would be harder to reproduce on a
> regular computer. I looked at all the parameters that I believe could
> affect this and believe for the most part the defaults are more than
> adequate. The main message queue never looked like it hit the high
> water mark but it did hit the lower one. So, I don't think messages
> were being dropped (not sure) or an overflow condition occurred.
>
> The processor is ARM-based and it is using Linux kernel 2.6.16.12 and
> compiled using GCC and the standard GNU C libraries version 3.4.5.
> Rsyslog source code is cross-compiled using the following configure
> line:
>
> ./configure --disable-zlib --disable-largefile
> --enable-share=yes
> --prefix=/
> --host=arm-unknown-linux-gnu
> ac_cv_func_malloc_0_nonnull=yes
> ac_cv_func_realloc_0_nonnull=yes
>
> ac_cv_func_lstat_dereferences_slashed_symlink=yes
> ac_cv_func_stat_empty_string_bug=no
> enable_debug=no
> enable_rtinst=no
>
> Lastly, the logtest was executed with just the "-s" parameter. It is a
> simple C file that I came up with.
>
> I took a look at the debug messages and it does not appear that new
> threads are created via calls to wtpStartWrkr in wtp.c.
>
> Any help I can bring to solve this issue, please let me know. I hope I
> am not doing anything wrong here.
>
> Thanks,
>
> Scott
> >
> > On Wed, Jul 2, 2008 at 12:04 PM, Scott Phuong
<mycleanjunk at gmail.com>
> wrote:
> >> Hi Rainer,
> >>
> >> Thanks for your reply. Looking at the default settings (from the
> >> online help's configuration page), they are what I wanted. The main
> >> messages queue is set to fix sized array with 1 worker thread
> created
> >> at maximum and action queues are direct mode which according to the
> >> queue document page, means that there will not be a worker thread
> >> created. Is my understanding correct? If yes, how do I quickly
> check
> >> without using the -d option if the defaults are set correctly? Or
> what
> >> do I look for in the debug messages that gets printed out to ensure
> >> this?
> >>
> >> You also mentioned that version 3.18.0 is probably going to be
> >> released as the stable version next week. I see on the webpage
there
> >> is a 3.17.4 and 3.17.5. Are these two versions similiar to 3.18.0?
> >>
> >> Also, how come I did not get your reply in my email inbox? My
> account
> >> settings look correct.
> >>
> >> Thanks,
> >>
> >> Scott Phuong
> >>
> >> As for the syslog buffer size, that applies to syslogd and does not
> >> apply to rsyslog.
> >>
> >>
> >>
> >> My configuration files do not change the Action queue or Worker
> queue
> >> parameters at all. Looking at
> >> On Wed, 2008-07-02 at 01:15 -0700, Scott Phuong wrote:
> >>> Hi,
> >>>
> >>> I have 3.16.2 which was recently released. I see that under
certain
> >>> conditions rsyslogd spawns a lot of threads:
> >>> 5949 root 11216 S rsyslogd
> >>> 5950 root 11216 S rsyslogd
> >>> 5951 root 11216 S rsyslogd
> >>> 5952 root 11216 S rsyslogd
> >>> 5953 root 11216 S rsyslogd
> >>> 5954 root 11216 S rsyslogd
> >>> 5985 root Z [rsyslogd]
> >>> 6445 root Z [rsyslogd]
> >>>
> >>> I had to kill the rsyslogd and restart it. The first invocation
had
> a
> >>> pid of 219 before it had to be killed. The second invocation of
pid
> >>> which you see above starts with 5949. The difference is the amount
> of
> >>> zombie threads that were invoked by rsyslogd before I had to kill
> the
> >>> first invocation of it.
> >>
> >> I have no explanation yet for the zombies. They should not happen
> and so
> >> far I have never seen them. We may need to go through a debug log
> (which
> >> will become very large) to find out what's going on.
> >>
> >>> The question is under what conditions does rsyslogd spawn a new
> >>> thread/process and why was it a zombie?
> >>
> >> Unfortunately, there is no quick answer. A quick one may be: when
it
> >> needs them, based on queue watermark settings and based on you
> >> configuration. But to really understand it, you need to read this
> doc:
> >>
> >> http://www.rsyslog.com/doc-queues.html
> >>
> >> The doc also describes all the knobs that you can use to control
> thread
> >> creation. There are many ;)
> >>
> >>> I am running rsyslogd in an
> >>> embedded environment and not a regular laptop/desktop.
> >>
> >> Interesting use case...
> >>
> >>> In addition, I
> >>> am using busybox and I believe the syslog buffer size is set to
> >>
> >> what do yo mean by "syslog buffer size"? The length of a receive
> buffer?
> >> It is 2K, thus single messages up to 2K are supported. It can be
> changed
> >> by modifying the MAXLINE define. Note that stock syslogd (and
> RFC3164)
> >> support only up to 1K.
> >>
> >>> something very low or perhaps none at all. Would this be a factor?
> >>> Furthermore, I ran rsyslogd with -c3 and also without -c3 and both
> >>> cases happen.
> >>
> >> The compatibility modes do not affect queue operation.
> >>
> >>> Are these issues already known and fixed in a later version?
Sorry,
> if
> >>> I am asking the same questions or have the same issues as previous
> >>> people but without the ability to search (or at least, I don't
know
> >>> how to) the archive, I don't know if my problem/questions has
> already
> >>> been seen and/or resolved.
> >>
> >> If we need to find out about the zombies, we need to move on to the
> >> current devel version. So I would give that a try in any case.
> 3.16.2
> >> will (most probably) be replaced by 3.18.0 (based on the current
> beta)
> >> next week. So I won't touch it any longer.
> >>
> >> Looking forward to your feedback,
> >> Rainer
> >>
> >>>
> >>> Thank you very much for your support.
> >>>
> >>> Scott
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>
> >
rsyslog threads questions [ In reply to ]
Hi Rainer,

Thanks!! I'll give this a try next week and let you know the results.
If it works, I think it is okay not to include in the upcoming v3
stable since it is due to release very soon. But if it does work, it
would be great to have it in the following stable release.

Scott

On Fri, Jul 4, 2008 at 1:31 AM, Rainer Gerhards
<rgerhards at hq.adiscon.com> wrote:
> Scott,
>
> I have rewritten the omusrmsg plugin. This is currently part of the
> development tree, but will become part of the (new) beta when we do a
> version number shift next week. I'd appreciate if you could give it a
> try. It is available here:
>
> http://download.rsyslog.com/download/rsyslog-3.19.9-pre1.tar.gz
>
> I am hesitant to put this in the upcoming new v3-stable. There were lots
> of changes and one rule for a stable is that there should be no non-fix
> type of changes for at least 4 weeks before it turns into a stable. It
> of course is debatable if the change I made is a fix or not. In some
> sense it is, but on the other hand it is a rewrite that changed a lot.
> So given the fact that nobody on a "regular" machine saw an issue the
> past year, I would not like to bring the new code directly into the
> stable. If that causes problems for you, you may want to try simply
> using the omusrmsg.c code together with the v3-stable. I haven't tried
> it, but it should work well. All changes were just to that file, and
> there are no dependencies to anything external.
>
> OK... but now let's see first if it fixes the issue ;)
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Scott Phuong
>> Sent: Thursday, July 03, 2008 2:58 AM
>> To: rsyslog at lists.adiscon.com
>> Subject: Re: [rsyslog] rsyslog threads questions
>>
>> Hi,
>>
>> I've attached four files. Two of which are debug dumps, one is the
>> conf file and the last one is a test case scenario that constantly
>> fails on my end. I hope this gives a little more information.
>> Furthermore, the dumps are from 3.17.5 which is the "closest" version
>> to 3.18.0 that I was able to find.
>>
>> Both failed scenarios occur when lots of messages were being flooded
>> to rsyslogd at a very fast rate (look at logtest.c) The
>> my_arm_rsyslog_suicide_debug.txt received a sigsegv fault while
>> my_arm_rsyslog_sh_cannot_fork caused so many "Z [rsyslogd]" threads
>> that it took up so much memory that executing any command as simple as
>> 'ls -l' would not work from the command line. I think the number of
>> threads grew as much as the number of messages. In the latter
>> scenario, after killing logtest.c, it didn't look like the those
>> zombies threads went away until I did a CTRL+C to the rsyslogd which
>> was running in the foreground since I use the "-dn" option.
>>
>> This is on an embedded system that runs significantly slower than a
>> desktop or laptop so maybe it would be harder to reproduce on a
>> regular computer. I looked at all the parameters that I believe could
>> affect this and believe for the most part the defaults are more than
>> adequate. The main message queue never looked like it hit the high
>> water mark but it did hit the lower one. So, I don't think messages
>> were being dropped (not sure) or an overflow condition occurred.
>>
>> The processor is ARM-based and it is using Linux kernel 2.6.16.12 and
>> compiled using GCC and the standard GNU C libraries version 3.4.5.
>> Rsyslog source code is cross-compiled using the following configure
>> line:
>>
>> ./configure --disable-zlib --disable-largefile
>> --enable-share=yes
>> --prefix=/
>> --host=arm-unknown-linux-gnu
>> ac_cv_func_malloc_0_nonnull=yes
>> ac_cv_func_realloc_0_nonnull=yes
>>
>> ac_cv_func_lstat_dereferences_slashed_symlink=yes
>> ac_cv_func_stat_empty_string_bug=no
>> enable_debug=no
>> enable_rtinst=no
>>
>> Lastly, the logtest was executed with just the "-s" parameter. It is a
>> simple C file that I came up with.
>>
>> I took a look at the debug messages and it does not appear that new
>> threads are created via calls to wtpStartWrkr in wtp.c.
>>
>> Any help I can bring to solve this issue, please let me know. I hope I
>> am not doing anything wrong here.
>>
>> Thanks,
>>
>> Scott
>> >
>> > On Wed, Jul 2, 2008 at 12:04 PM, Scott Phuong
> <mycleanjunk at gmail.com>
>> wrote:
>> >> Hi Rainer,
>> >>
>> >> Thanks for your reply. Looking at the default settings (from the
>> >> online help's configuration page), they are what I wanted. The main
>> >> messages queue is set to fix sized array with 1 worker thread
>> created
>> >> at maximum and action queues are direct mode which according to the
>> >> queue document page, means that there will not be a worker thread
>> >> created. Is my understanding correct? If yes, how do I quickly
>> check
>> >> without using the -d option if the defaults are set correctly? Or
>> what
>> >> do I look for in the debug messages that gets printed out to ensure
>> >> this?
>> >>
>> >> You also mentioned that version 3.18.0 is probably going to be
>> >> released as the stable version next week. I see on the webpage
> there
>> >> is a 3.17.4 and 3.17.5. Are these two versions similiar to 3.18.0?
>> >>
>> >> Also, how come I did not get your reply in my email inbox? My
>> account
>> >> settings look correct.
>> >>
>> >> Thanks,
>> >>
>> >> Scott Phuong
>> >>
>> >> As for the syslog buffer size, that applies to syslogd and does not
>> >> apply to rsyslog.
>> >>
>> >>
>> >>
>> >> My configuration files do not change the Action queue or Worker
>> queue
>> >> parameters at all. Looking at
>> >> On Wed, 2008-07-02 at 01:15 -0700, Scott Phuong wrote:
>> >>> Hi,
>> >>>
>> >>> I have 3.16.2 which was recently released. I see that under
> certain
>> >>> conditions rsyslogd spawns a lot of threads:
>> >>> 5949 root 11216 S rsyslogd
>> >>> 5950 root 11216 S rsyslogd
>> >>> 5951 root 11216 S rsyslogd
>> >>> 5952 root 11216 S rsyslogd
>> >>> 5953 root 11216 S rsyslogd
>> >>> 5954 root 11216 S rsyslogd
>> >>> 5985 root Z [rsyslogd]
>> >>> 6445 root Z [rsyslogd]
>> >>>
>> >>> I had to kill the rsyslogd and restart it. The first invocation
> had
>> a
>> >>> pid of 219 before it had to be killed. The second invocation of
> pid
>> >>> which you see above starts with 5949. The difference is the amount
>> of
>> >>> zombie threads that were invoked by rsyslogd before I had to kill
>> the
>> >>> first invocation of it.
>> >>
>> >> I have no explanation yet for the zombies. They should not happen
>> and so
>> >> far I have never seen them. We may need to go through a debug log
>> (which
>> >> will become very large) to find out what's going on.
>> >>
>> >>> The question is under what conditions does rsyslogd spawn a new
>> >>> thread/process and why was it a zombie?
>> >>
>> >> Unfortunately, there is no quick answer. A quick one may be: when
> it
>> >> needs them, based on queue watermark settings and based on you
>> >> configuration. But to really understand it, you need to read this
>> doc:
>> >>
>> >> http://www.rsyslog.com/doc-queues.html
>> >>
>> >> The doc also describes all the knobs that you can use to control
>> thread
>> >> creation. There are many ;)
>> >>
>> >>> I am running rsyslogd in an
>> >>> embedded environment and not a regular laptop/desktop.
>> >>
>> >> Interesting use case...
>> >>
>> >>> In addition, I
>> >>> am using busybox and I believe the syslog buffer size is set to
>> >>
>> >> what do yo mean by "syslog buffer size"? The length of a receive
>> buffer?
>> >> It is 2K, thus single messages up to 2K are supported. It can be
>> changed
>> >> by modifying the MAXLINE define. Note that stock syslogd (and
>> RFC3164)
>> >> support only up to 1K.
>> >>
>> >>> something very low or perhaps none at all. Would this be a factor?
>> >>> Furthermore, I ran rsyslogd with -c3 and also without -c3 and both
>> >>> cases happen.
>> >>
>> >> The compatibility modes do not affect queue operation.
>> >>
>> >>> Are these issues already known and fixed in a later version?
> Sorry,
>> if
>> >>> I am asking the same questions or have the same issues as previous
>> >>> people but without the ability to search (or at least, I don't
> know
>> >>> how to) the archive, I don't know if my problem/questions has
>> already
>> >>> been seen and/or resolved.
>> >>
>> >> If we need to find out about the zombies, we need to move on to the
>> >> current devel version. So I would give that a try in any case.
>> 3.16.2
>> >> will (most probably) be replaced by 3.18.0 (based on the current
>> beta)
>> >> next week. So I won't touch it any longer.
>> >>
>> >> Looking forward to your feedback,
>> >> Rainer
>> >>
>> >>>
>> >>> Thank you very much for your support.
>> >>>
>> >>> Scott
>> >>> _______________________________________________
>> >>> rsyslog mailing list
>> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>
>> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>
rsyslog threads questions [ In reply to ]
Hi Scott,

just for your understanding: once a release is stable, it will only
receive fixes. So the fix we are looking it needs to become part of the
devel, be migrated to beta and then be migrated to stable. In general,
one migration step takes between 4 and 12 weeks, depending on what was
done. So far, the average seems on the average, that is two month. So
two cycles mean roughly four month. HOWEVER, I intend to put it into the
next beta, which will bring us to just one cycle. So it would be
available in roughly two month.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Scott Phuong
> Sent: Friday, July 04, 2008 12:32 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog threads questions
>
> Hi Rainer,
>
> Thanks!! I'll give this a try next week and let you know the results.
> If it works, I think it is okay not to include in the upcoming v3
> stable since it is due to release very soon. But if it does work, it
> would be great to have it in the following stable release.
>
> Scott
>
> On Fri, Jul 4, 2008 at 1:31 AM, Rainer Gerhards
> <rgerhards at hq.adiscon.com> wrote:
> > Scott,
> >
> > I have rewritten the omusrmsg plugin. This is currently part of the
> > development tree, but will become part of the (new) beta when we do
a
> > version number shift next week. I'd appreciate if you could give it
a
> > try. It is available here:
> >
> > http://download.rsyslog.com/download/rsyslog-3.19.9-pre1.tar.gz
> >
> > I am hesitant to put this in the upcoming new v3-stable. There were
> lots
> > of changes and one rule for a stable is that there should be no non-
> fix
> > type of changes for at least 4 weeks before it turns into a stable.
> It
> > of course is debatable if the change I made is a fix or not. In some
> > sense it is, but on the other hand it is a rewrite that changed a
> lot.
> > So given the fact that nobody on a "regular" machine saw an issue
the
> > past year, I would not like to bring the new code directly into the
> > stable. If that causes problems for you, you may want to try simply
> > using the omusrmsg.c code together with the v3-stable. I haven't
> tried
> > it, but it should work well. All changes were just to that file, and
> > there are no dependencies to anything external.
> >
> > OK... but now let's see first if it fixes the issue ;)
> >
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Scott Phuong
> >> Sent: Thursday, July 03, 2008 2:58 AM
> >> To: rsyslog at lists.adiscon.com
> >> Subject: Re: [rsyslog] rsyslog threads questions
> >>
> >> Hi,
> >>
> >> I've attached four files. Two of which are debug dumps, one is the
> >> conf file and the last one is a test case scenario that constantly
> >> fails on my end. I hope this gives a little more information.
> >> Furthermore, the dumps are from 3.17.5 which is the "closest"
> version
> >> to 3.18.0 that I was able to find.
> >>
> >> Both failed scenarios occur when lots of messages were being
flooded
> >> to rsyslogd at a very fast rate (look at logtest.c) The
> >> my_arm_rsyslog_suicide_debug.txt received a sigsegv fault while
> >> my_arm_rsyslog_sh_cannot_fork caused so many "Z [rsyslogd]"
> threads
> >> that it took up so much memory that executing any command as simple
> as
> >> 'ls -l' would not work from the command line. I think the number of
> >> threads grew as much as the number of messages. In the latter
> >> scenario, after killing logtest.c, it didn't look like the those
> >> zombies threads went away until I did a CTRL+C to the rsyslogd
which
> >> was running in the foreground since I use the "-dn" option.
> >>
> >> This is on an embedded system that runs significantly slower than a
> >> desktop or laptop so maybe it would be harder to reproduce on a
> >> regular computer. I looked at all the parameters that I believe
> could
> >> affect this and believe for the most part the defaults are more
than
> >> adequate. The main message queue never looked like it hit the high
> >> water mark but it did hit the lower one. So, I don't think messages
> >> were being dropped (not sure) or an overflow condition occurred.
> >>
> >> The processor is ARM-based and it is using Linux kernel 2.6.16.12
> and
> >> compiled using GCC and the standard GNU C libraries version 3.4.5.
> >> Rsyslog source code is cross-compiled using the following configure
> >> line:
> >>
> >> ./configure --disable-zlib --disable-largefile
> >> --enable-share=yes
> >> --prefix=/
> >> --host=arm-unknown-linux-gnu
> >> ac_cv_func_malloc_0_nonnull=yes
> >> ac_cv_func_realloc_0_nonnull=yes
> >>
> >> ac_cv_func_lstat_dereferences_slashed_symlink=yes
> >> ac_cv_func_stat_empty_string_bug=no
> >> enable_debug=no
> >> enable_rtinst=no
> >>
> >> Lastly, the logtest was executed with just the "-s" parameter. It
is
> a
> >> simple C file that I came up with.
> >>
> >> I took a look at the debug messages and it does not appear that new
> >> threads are created via calls to wtpStartWrkr in wtp.c.
> >>
> >> Any help I can bring to solve this issue, please let me know. I
hope
> I
> >> am not doing anything wrong here.
> >>
> >> Thanks,
> >>
> >> Scott
> >> >
> >> > On Wed, Jul 2, 2008 at 12:04 PM, Scott Phuong
> > <mycleanjunk at gmail.com>
> >> wrote:
> >> >> Hi Rainer,
> >> >>
> >> >> Thanks for your reply. Looking at the default settings (from
the
> >> >> online help's configuration page), they are what I wanted. The
> main
> >> >> messages queue is set to fix sized array with 1 worker thread
> >> created
> >> >> at maximum and action queues are direct mode which according to
> the
> >> >> queue document page, means that there will not be a worker
thread
> >> >> created. Is my understanding correct? If yes, how do I quickly
> >> check
> >> >> without using the -d option if the defaults are set correctly?
Or
> >> what
> >> >> do I look for in the debug messages that gets printed out to
> ensure
> >> >> this?
> >> >>
> >> >> You also mentioned that version 3.18.0 is probably going to be
> >> >> released as the stable version next week. I see on the webpage
> > there
> >> >> is a 3.17.4 and 3.17.5. Are these two versions similiar to
> 3.18.0?
> >> >>
> >> >> Also, how come I did not get your reply in my email inbox? My
> >> account
> >> >> settings look correct.
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Scott Phuong
> >> >>
> >> >> As for the syslog buffer size, that applies to syslogd and does
> not
> >> >> apply to rsyslog.
> >> >>
> >> >>
> >> >>
> >> >> My configuration files do not change the Action queue or Worker
> >> queue
> >> >> parameters at all. Looking at
> >> >> On Wed, 2008-07-02 at 01:15 -0700, Scott Phuong wrote:
> >> >>> Hi,
> >> >>>
> >> >>> I have 3.16.2 which was recently released. I see that under
> > certain
> >> >>> conditions rsyslogd spawns a lot of threads:
> >> >>> 5949 root 11216 S rsyslogd
> >> >>> 5950 root 11216 S rsyslogd
> >> >>> 5951 root 11216 S rsyslogd
> >> >>> 5952 root 11216 S rsyslogd
> >> >>> 5953 root 11216 S rsyslogd
> >> >>> 5954 root 11216 S rsyslogd
> >> >>> 5985 root Z [rsyslogd]
> >> >>> 6445 root Z [rsyslogd]
> >> >>>
> >> >>> I had to kill the rsyslogd and restart it. The first invocation
> > had
> >> a
> >> >>> pid of 219 before it had to be killed. The second invocation of
> > pid
> >> >>> which you see above starts with 5949. The difference is the
> amount
> >> of
> >> >>> zombie threads that were invoked by rsyslogd before I had to
> kill
> >> the
> >> >>> first invocation of it.
> >> >>
> >> >> I have no explanation yet for the zombies. They should not
happen
> >> and so
> >> >> far I have never seen them. We may need to go through a debug
log
> >> (which
> >> >> will become very large) to find out what's going on.
> >> >>
> >> >>> The question is under what conditions does rsyslogd spawn a new
> >> >>> thread/process and why was it a zombie?
> >> >>
> >> >> Unfortunately, there is no quick answer. A quick one may be:
when
> > it
> >> >> needs them, based on queue watermark settings and based on you
> >> >> configuration. But to really understand it, you need to read
this
> >> doc:
> >> >>
> >> >> http://www.rsyslog.com/doc-queues.html
> >> >>
> >> >> The doc also describes all the knobs that you can use to control
> >> thread
> >> >> creation. There are many ;)
> >> >>
> >> >>> I am running rsyslogd in an
> >> >>> embedded environment and not a regular laptop/desktop.
> >> >>
> >> >> Interesting use case...
> >> >>
> >> >>> In addition, I
> >> >>> am using busybox and I believe the syslog buffer size is set to
> >> >>
> >> >> what do yo mean by "syslog buffer size"? The length of a receive
> >> buffer?
> >> >> It is 2K, thus single messages up to 2K are supported. It can be
> >> changed
> >> >> by modifying the MAXLINE define. Note that stock syslogd (and
> >> RFC3164)
> >> >> support only up to 1K.
> >> >>
> >> >>> something very low or perhaps none at all. Would this be a
> factor?
> >> >>> Furthermore, I ran rsyslogd with -c3 and also without -c3 and
> both
> >> >>> cases happen.
> >> >>
> >> >> The compatibility modes do not affect queue operation.
> >> >>
> >> >>> Are these issues already known and fixed in a later version?
> > Sorry,
> >> if
> >> >>> I am asking the same questions or have the same issues as
> previous
> >> >>> people but without the ability to search (or at least, I don't
> > know
> >> >>> how to) the archive, I don't know if my problem/questions has
> >> already
> >> >>> been seen and/or resolved.
> >> >>
> >> >> If we need to find out about the zombies, we need to move on to
> the
> >> >> current devel version. So I would give that a try in any case.
> >> 3.16.2
> >> >> will (most probably) be replaced by 3.18.0 (based on the current
> >> beta)
> >> >> next week. So I won't touch it any longer.
> >> >>
> >> >> Looking forward to your feedback,
> >> >> Rainer
> >> >>
> >> >>>
> >> >>> Thank you very much for your support.
> >> >>>
> >> >>> Scott
> >> >>> _______________________________________________
> >> >>> rsyslog mailing list
> >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> >>
> >> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
rsyslog threads questions [ In reply to ]
Hi Rainer,

Thanks for the information. I didn't know that.

Scott

On Fri, Jul 4, 2008 at 3:35 AM, Rainer Gerhards
<rgerhards at hq.adiscon.com> wrote:
> Hi Scott,
>
> just for your understanding: once a release is stable, it will only
> receive fixes. So the fix we are looking it needs to become part of the
> devel, be migrated to beta and then be migrated to stable. In general,
> one migration step takes between 4 and 12 weeks, depending on what was
> done. So far, the average seems on the average, that is two month. So
> two cycles mean roughly four month. HOWEVER, I intend to put it into the
> next beta, which will bring us to just one cycle. So it would be
> available in roughly two month.
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Scott Phuong
>> Sent: Friday, July 04, 2008 12:32 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] rsyslog threads questions
>>
>> Hi Rainer,
>>
>> Thanks!! I'll give this a try next week and let you know the results.
>> If it works, I think it is okay not to include in the upcoming v3
>> stable since it is due to release very soon. But if it does work, it
>> would be great to have it in the following stable release.
>>
>> Scott
>>
>> On Fri, Jul 4, 2008 at 1:31 AM, Rainer Gerhards
>> <rgerhards at hq.adiscon.com> wrote:
>> > Scott,
>> >
>> > I have rewritten the omusrmsg plugin. This is currently part of the
>> > development tree, but will become part of the (new) beta when we do
> a
>> > version number shift next week. I'd appreciate if you could give it
> a
>> > try. It is available here:
>> >
>> > http://download.rsyslog.com/download/rsyslog-3.19.9-pre1.tar.gz
>> >
>> > I am hesitant to put this in the upcoming new v3-stable. There were
>> lots
>> > of changes and one rule for a stable is that there should be no non-
>> fix
>> > type of changes for at least 4 weeks before it turns into a stable.
>> It
>> > of course is debatable if the change I made is a fix or not. In some
>> > sense it is, but on the other hand it is a rewrite that changed a
>> lot.
>> > So given the fact that nobody on a "regular" machine saw an issue
> the
>> > past year, I would not like to bring the new code directly into the
>> > stable. If that causes problems for you, you may want to try simply
>> > using the omusrmsg.c code together with the v3-stable. I haven't
>> tried
>> > it, but it should work well. All changes were just to that file, and
>> > there are no dependencies to anything external.
>> >
>> > OK... but now let's see first if it fixes the issue ;)
>> >
>> > Rainer
>> >
>> >> -----Original Message-----
>> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> >> bounces at lists.adiscon.com] On Behalf Of Scott Phuong
>> >> Sent: Thursday, July 03, 2008 2:58 AM
>> >> To: rsyslog at lists.adiscon.com
>> >> Subject: Re: [rsyslog] rsyslog threads questions
>> >>
>> >> Hi,
>> >>
>> >> I've attached four files. Two of which are debug dumps, one is the
>> >> conf file and the last one is a test case scenario that constantly
>> >> fails on my end. I hope this gives a little more information.
>> >> Furthermore, the dumps are from 3.17.5 which is the "closest"
>> version
>> >> to 3.18.0 that I was able to find.
>> >>
>> >> Both failed scenarios occur when lots of messages were being
> flooded
>> >> to rsyslogd at a very fast rate (look at logtest.c) The
>> >> my_arm_rsyslog_suicide_debug.txt received a sigsegv fault while
>> >> my_arm_rsyslog_sh_cannot_fork caused so many "Z [rsyslogd]"
>> threads
>> >> that it took up so much memory that executing any command as simple
>> as
>> >> 'ls -l' would not work from the command line. I think the number of
>> >> threads grew as much as the number of messages. In the latter
>> >> scenario, after killing logtest.c, it didn't look like the those
>> >> zombies threads went away until I did a CTRL+C to the rsyslogd
> which
>> >> was running in the foreground since I use the "-dn" option.
>> >>
>> >> This is on an embedded system that runs significantly slower than a
>> >> desktop or laptop so maybe it would be harder to reproduce on a
>> >> regular computer. I looked at all the parameters that I believe
>> could
>> >> affect this and believe for the most part the defaults are more
> than
>> >> adequate. The main message queue never looked like it hit the high
>> >> water mark but it did hit the lower one. So, I don't think messages
>> >> were being dropped (not sure) or an overflow condition occurred.
>> >>
>> >> The processor is ARM-based and it is using Linux kernel 2.6.16.12
>> and
>> >> compiled using GCC and the standard GNU C libraries version 3.4.5.
>> >> Rsyslog source code is cross-compiled using the following configure
>> >> line:
>> >>
>> >> ./configure --disable-zlib --disable-largefile
>> >> --enable-share=yes
>> >> --prefix=/
>> >> --host=arm-unknown-linux-gnu
>> >> ac_cv_func_malloc_0_nonnull=yes
>> >> ac_cv_func_realloc_0_nonnull=yes
>> >>
>> >> ac_cv_func_lstat_dereferences_slashed_symlink=yes
>> >> ac_cv_func_stat_empty_string_bug=no
>> >> enable_debug=no
>> >> enable_rtinst=no
>> >>
>> >> Lastly, the logtest was executed with just the "-s" parameter. It
> is
>> a
>> >> simple C file that I came up with.
>> >>
>> >> I took a look at the debug messages and it does not appear that new
>> >> threads are created via calls to wtpStartWrkr in wtp.c.
>> >>
>> >> Any help I can bring to solve this issue, please let me know. I
> hope
>> I
>> >> am not doing anything wrong here.
>> >>
>> >> Thanks,
>> >>
>> >> Scott
>> >> >
>> >> > On Wed, Jul 2, 2008 at 12:04 PM, Scott Phuong
>> > <mycleanjunk at gmail.com>
>> >> wrote:
>> >> >> Hi Rainer,
>> >> >>
>> >> >> Thanks for your reply. Looking at the default settings (from
> the
>> >> >> online help's configuration page), they are what I wanted. The
>> main
>> >> >> messages queue is set to fix sized array with 1 worker thread
>> >> created
>> >> >> at maximum and action queues are direct mode which according to
>> the
>> >> >> queue document page, means that there will not be a worker
> thread
>> >> >> created. Is my understanding correct? If yes, how do I quickly
>> >> check
>> >> >> without using the -d option if the defaults are set correctly?
> Or
>> >> what
>> >> >> do I look for in the debug messages that gets printed out to
>> ensure
>> >> >> this?
>> >> >>
>> >> >> You also mentioned that version 3.18.0 is probably going to be
>> >> >> released as the stable version next week. I see on the webpage
>> > there
>> >> >> is a 3.17.4 and 3.17.5. Are these two versions similiar to
>> 3.18.0?
>> >> >>
>> >> >> Also, how come I did not get your reply in my email inbox? My
>> >> account
>> >> >> settings look correct.
>> >> >>
>> >> >> Thanks,
>> >> >>
>> >> >> Scott Phuong
>> >> >>
>> >> >> As for the syslog buffer size, that applies to syslogd and does
>> not
>> >> >> apply to rsyslog.
>> >> >>
>> >> >>
>> >> >>
>> >> >> My configuration files do not change the Action queue or Worker
>> >> queue
>> >> >> parameters at all. Looking at
>> >> >> On Wed, 2008-07-02 at 01:15 -0700, Scott Phuong wrote:
>> >> >>> Hi,
>> >> >>>
>> >> >>> I have 3.16.2 which was recently released. I see that under
>> > certain
>> >> >>> conditions rsyslogd spawns a lot of threads:
>> >> >>> 5949 root 11216 S rsyslogd
>> >> >>> 5950 root 11216 S rsyslogd
>> >> >>> 5951 root 11216 S rsyslogd
>> >> >>> 5952 root 11216 S rsyslogd
>> >> >>> 5953 root 11216 S rsyslogd
>> >> >>> 5954 root 11216 S rsyslogd
>> >> >>> 5985 root Z [rsyslogd]
>> >> >>> 6445 root Z [rsyslogd]
>> >> >>>
>> >> >>> I had to kill the rsyslogd and restart it. The first invocation
>> > had
>> >> a
>> >> >>> pid of 219 before it had to be killed. The second invocation of
>> > pid
>> >> >>> which you see above starts with 5949. The difference is the
>> amount
>> >> of
>> >> >>> zombie threads that were invoked by rsyslogd before I had to
>> kill
>> >> the
>> >> >>> first invocation of it.
>> >> >>
>> >> >> I have no explanation yet for the zombies. They should not
> happen
>> >> and so
>> >> >> far I have never seen them. We may need to go through a debug
> log
>> >> (which
>> >> >> will become very large) to find out what's going on.
>> >> >>
>> >> >>> The question is under what conditions does rsyslogd spawn a new
>> >> >>> thread/process and why was it a zombie?
>> >> >>
>> >> >> Unfortunately, there is no quick answer. A quick one may be:
> when
>> > it
>> >> >> needs them, based on queue watermark settings and based on you
>> >> >> configuration. But to really understand it, you need to read
> this
>> >> doc:
>> >> >>
>> >> >> http://www.rsyslog.com/doc-queues.html
>> >> >>
>> >> >> The doc also describes all the knobs that you can use to control
>> >> thread
>> >> >> creation. There are many ;)
>> >> >>
>> >> >>> I am running rsyslogd in an
>> >> >>> embedded environment and not a regular laptop/desktop.
>> >> >>
>> >> >> Interesting use case...
>> >> >>
>> >> >>> In addition, I
>> >> >>> am using busybox and I believe the syslog buffer size is set to
>> >> >>
>> >> >> what do yo mean by "syslog buffer size"? The length of a receive
>> >> buffer?
>> >> >> It is 2K, thus single messages up to 2K are supported. It can be
>> >> changed
>> >> >> by modifying the MAXLINE define. Note that stock syslogd (and
>> >> RFC3164)
>> >> >> support only up to 1K.
>> >> >>
>> >> >>> something very low or perhaps none at all. Would this be a
>> factor?
>> >> >>> Furthermore, I ran rsyslogd with -c3 and also without -c3 and
>> both
>> >> >>> cases happen.
>> >> >>
>> >> >> The compatibility modes do not affect queue operation.
>> >> >>
>> >> >>> Are these issues already known and fixed in a later version?
>> > Sorry,
>> >> if
>> >> >>> I am asking the same questions or have the same issues as
>> previous
>> >> >>> people but without the ability to search (or at least, I don't
>> > know
>> >> >>> how to) the archive, I don't know if my problem/questions has
>> >> already
>> >> >>> been seen and/or resolved.
>> >> >>
>> >> >> If we need to find out about the zombies, we need to move on to
>> the
>> >> >> current devel version. So I would give that a try in any case.
>> >> 3.16.2
>> >> >> will (most probably) be replaced by 3.18.0 (based on the current
>> >> beta)
>> >> >> next week. So I won't touch it any longer.
>> >> >>
>> >> >> Looking forward to your feedback,
>> >> >> Rainer
>> >> >>
>> >> >>>
>> >> >>> Thank you very much for your support.
>> >> >>>
>> >> >>> Scott
>> >> >>> _______________________________________________
>> >> >>> rsyslog mailing list
>> >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >>
>> >> >
>> > _______________________________________________
>> > rsyslog mailing list
>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>