Mailing List Archive

On Tue, Sep 25, 2007 at 11:46:14AM +0200, Rainer Gerhards wrote:
> Hi all,
>
> Rsyslog 1.19.7 (the official one) has been released today. It contains
> some code updated compared to the version from yesterday, so please
> download it if you already did.
>
> Rsyslog 1.19.7 is a code cleanup, bug fixing and maintenance release.
> Its primary goal is enhanced stability. Feature-wise, it offers support
> of dropping NUL characters, which are often found at the end of
> invalidly formatted syslog messages. Upgrading to this release is
> recommended for all users.

You said you don't support rfc3195d at the moment, but anyway:

rfc3195d.c:108: error: 'errStr' undeclared (first use in this
function)

Oops.. looks like I need to check my build procedure, obviously it
wasn't build over here...

Thanks for the report, it's fixed now.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Andrew Pantyukhin
> Sent: Tuesday, September 25, 2007 12:42 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 1.19.7 released
>
> On Tue, Sep 25, 2007 at 11:46:14AM +0200, Rainer Gerhards wrote:
> > Hi all,
> >
> > Rsyslog 1.19.7 (the official one) has been released today. It
> contains
> > some code updated compared to the version from yesterday, so please
> > download it if you already did.
> >
> > Rsyslog 1.19.7 is a code cleanup, bug fixing and maintenance
release.
> > Its primary goal is enhanced stability. Feature-wise, it offers
> support
> > of dropping NUL characters, which are often found at the end of
> > invalidly formatted syslog messages. Upgrading to this release is
> > recommended for all users.
>
> You said you don't support rfc3195d at the moment, but anyway:
>
> rfc3195d.c:108: error: 'errStr' undeclared (first use in this
> function)
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog

Am Dienstag, den 25.09.2007, 11:46 +0200 schrieb Rainer Gerhards:
[...]
> As always, feedback is appreciated. Feedback is especially appreciated
> from those that have experienced the segfault problem in the past.
Hi all,

I'd like to apologize, since it was me who reported to Rainer that the
segfault problem seems to have gone with the new release and that everything
seems to be better now before he released 1.19.7.

Meanwhile I realised that while the rsyslog-Daemon indeed ran without any
segfaults, however, it didn't log anything sent to it via UDP. So my segfault
problem has been replaced by a new one. ;-) So everybody should be very
careful with this update, especially those who previously encountered the
segfault problem.

--
CU,
Patrick.

Patrick,

you mean nothing was logged via UDP while local domain sockets and/or
TCP still worked? I have to admit I am a bit puzzled...

Do you have a debug log at hand?

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Patrick von der Hagen
> Sent: Tuesday, September 25, 2007 1:55 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 1.19.7 released
>
> Am Dienstag, den 25.09.2007, 11:46 +0200 schrieb Rainer Gerhards:
> [...]
> > As always, feedback is appreciated. Feedback is especially
> appreciated
> > from those that have experienced the segfault problem in the past.
> Hi all,
>
> I'd like to apologize, since it was me who reported to Rainer that the
> segfault problem seems to have gone with the new release and that
> everything
> seems to be better now before he released 1.19.7.
>
> Meanwhile I realised that while the rsyslog-Daemon indeed ran without
> any
> segfaults, however, it didn't log anything sent to it via UDP. So my
> segfault
> problem has been replaced by a new one. ;-) So everybody should be
very
> careful with this update, especially those who previously encountered
> the
> segfault problem.
>
> --
> CU,
> Patrick.
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog

Am Dienstag, den 25.09.2007, 13:57 +0200 schrieb Rainer Gerhards:
> Patrick,
>
> you mean nothing was logged via UDP while local domain sockets and/or
> TCP still worked? I have to admit I am a bit puzzled...
I only use rsyslog for UDP-logging, so I can't comment on TCP or local
domain sockets. I usually run rsyslogd with "-r514" or "-4 -r514". The
only log-entries relate to startup or shutdown of rsyslog itself.


> Do you have a debug log at hand?
Let's start with compiler warnings. They are identical to 1.19.6, so
probably not related to the changed behaviour.



syslog.c: In function ?writeSyslogV?:
syslog.c:98: warning: function might be possible candidate for ?printf?
format attribute
syslog.c:98: warning: function might be possible candidate for ?printf?
format attribute

template.c: In function ?tplPrintList?:
template.c:927: warning: cast from pointer to integer of different size

modules.c: In function ?modPrintList?:
modules.c:309: warning: cast from pointer to integer of different size
modules.c:310: warning: cast from pointer to integer of different size
modules.c:311: warning: cast from pointer to integer of different size
modules.c:312: warning: cast from pointer to integer of different size
modules.c:313: warning: cast from pointer to integer of different size

cfsysline.c: In function ?dbgPrintCfSysLineHandlers?:
cfsysline.c:710: warning: cast from pointer to integer of different size
cfsysline.c:711: warning: cast from pointer to integer of different size

action.c: In function ?actionDbgPrint?:
action.c:174: warning: cast from pointer to integer of different size


If I run "rsyslogd -r514 -4 -d" it prints something like this:
Successful select, descriptor count = 1, Activity on: 6
-1431504256: Listening on UDP syslogd socket 6 (IPv4/port 514).
-1431504256: ----------------------------------------
-1431504256: Calling select, active file descriptors (max 6): 3 6
-1431504256:
Successful select, descriptor count = 1, Activity on: 6
-1431504256: Listening on UDP syslogd socket 6 (IPv4/port 514).
-1431504256: ----------------------------------------
-1431504256: Calling select, active file descriptors (max 6): 3 6
-1431504256:
Successful select, descriptor count = 1, Activity on: 6
-1431504256: Listening on UDP syslogd socket 6 (IPv4/port 514).
-1431504256: ----------------------------------------
-1431504256: Calling select, active file descriptors (max 6): 3 6
-1431504256:

over and over again.

I'll sent you a complete output off-list.

--
CU,
Patrick.

Hi Rainer,

> Hi all,
>
> Rsyslog 1.19.7 (the official one) has been released today. It
> contains some code updated compared to the version from yesterday,
> so please download it if you already did.

I've just downloaded and installed this release, and although I've never
experienced the segfault issue (rsyslog continually runs fine for me), when I
installed this version and then did a:

# service rsyslog restart

I got this in my messages file:

Sep 25 22:41:46 server kernel: Kernel logging (proc) stopped.
Sep 25 22:41:46 server kernel: Kernel log daemon terminating.
Sep 25 22:41:47 server rsyslog: rklogd shutdown succeeded
Sep 25 22:41:47 server rsyslogd: [origin software="rsyslogd"
swVersion="1.19.7" x-pid="12176"] exiting on signal 15.
Sep 25 22:44:08 server rsyslogd: [origin software="rsyslogd"
swVersion="1.19.7" x-pid="10430"][x-configInfo udpRecep
tion="No" udpPort="514" tcpReception="No" tcpPort="0"] restart
Sep 25 22:44:08 server rsyslog: rsyslogd startup succeeded
Sep 25 22:44:08 server kernel: rklogd 1.19.7, log source = /proc/kmsg started.
Sep 25 22:44:08 server kernel: rsyslogd[8110]: segfault at 000000000000000a
rip 00000038c7568678 rsp 0000007fbfffdc2
0 error 4
Sep 25 22:44:08 server rsyslog: rklogd startup succeeded
Sep 25 22:44:08 server rsyslog: rsyslogd shutdown succeeded

The above machine is a DL380 G4 with one CPU running hypthreaded. On another
machine that's the same as that:

Sep 25 22:42:24 server2 kernel: Kernel logging (proc) stopped.
Sep 25 22:42:24 server2 kernel: Kernel log daemon terminating.
Sep 25 22:42:25 server2 rsyslog: rklogd shutdown succeeded
Sep 25 22:42:25 server2 rsyslogd: [origin software="rsyslogd"
swVersion="1.19.7" x-pid="28497"] exiting on signal 15
.
Sep 25 22:43:43 server2 rsyslogd: [origin software="rsyslogd"
swVersion="1.19.7" x-pid="16598"][x-configInfo udpRece
ption="No" udpPort="514" tcpReception="No" tcpPort="0"] restart
Sep 25 22:43:43 server2 rsyslog: rsyslogd startup succeeded
Sep 25 22:43:43 server2 kernel: rklogd 1.19.7, log source = /proc/kmsg started.
Sep 25 22:43:43 server2 kernel: rsyslogd[15998]: segfault at 000000000000000a
rip 00000034d2868678 rsp 0000007fbfffd
c20 error 4
Sep 25 22:43:43 server2 rsyslog: rklogd startup succeeded

Those two machines above are also running 64bit Linux.

Using the same process on other 32 bit linux servers (DL360's), no segfaults
occur.

Note that I have never (ever) had rsyslog crash on me while it's running, and
I currently run it on 6 servers to do the MySQL logging.

Regards,

Michael.

> Rsyslog 1.19.7 is a code cleanup, bug fixing and maintenance release.
> Its primary goal is enhanced stability. Feature-wise, it offers support
> of dropping NUL characters, which are often found at the end of
> invalidly formatted syslog messages. Upgrading to this release is
> recommended for all users.
>
> Changelog:
>
> http://www.rsyslog.com/Article129.phtml
>
> Download:
>
> http://www.rsyslog.com/Downloads-req-getit-lid-59.phtml
>
> PLEASE NOTE: the rsyslog site is currently migrated to a new
> machine. In case you receive a temporary page, your DNS still caches
> the old machine's address.
>
> As always, feedback is appreciated. Feedback is especially
> appreciated from those that have experienced the segfault problem in
> the past.
>
> Rainer Gerhards
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
------- End of Original Message -------

Sorry Michael,

I just noticed that I had not replied. I looked at the code, but there
is too few information to track the source code location down. What I
currently assume is

a) that we still have a segfault issue (obviously ;))
b) the segfault is a moving target

The segfault occurs due to memory corruption. I think (and have often
seen) that you do not notice the corruption until it happens to hit some
really important piece of memory. So in many cases, it will overwrite
something that nobody cares about. Only if some specific structures are
hit, a malfunction manifests. This is part of the problem to reproducing
it. What is hit, depends on the memory layout. A new version leads to
different memory layout.

I assume this is what has happened here: you did not experience the bug
before, because it hit some memory that wasn't really relevant. With the
new versions memory layout, it hits somewhere where it hurts... At least
this is my best guess.

I'd appreciate if you could try out the 1.19.8 release. I suspect it
will also segfault. If it does, it would be great if you could run it in
debug mode (-d -n) and send me the output before it segfault (maybe 200
lines or so).

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Mansour
> Sent: Tuesday, September 25, 2007 3:41 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 1.19.7 released
>
> Hi Rainer,
>
> > Hi all,
> >
> > Rsyslog 1.19.7 (the official one) has been released today. It
> > contains some code updated compared to the version from yesterday,
> > so please download it if you already did.
>
> I've just downloaded and installed this release, and although I've
> never
> experienced the segfault issue (rsyslog continually runs fine for me),
> when I
> installed this version and then did a:
>
> # service rsyslog restart
>
> I got this in my messages file:
>
> Sep 25 22:41:46 server kernel: Kernel logging (proc) stopped.
> Sep 25 22:41:46 server kernel: Kernel log daemon terminating.
> Sep 25 22:41:47 server rsyslog: rklogd shutdown succeeded
> Sep 25 22:41:47 server rsyslogd: [origin software="rsyslogd"
> swVersion="1.19.7" x-pid="12176"] exiting on signal 15.
> Sep 25 22:44:08 server rsyslogd: [origin software="rsyslogd"
> swVersion="1.19.7" x-pid="10430"][x-configInfo udpRecep
> tion="No" udpPort="514" tcpReception="No" tcpPort="0"] restart
> Sep 25 22:44:08 server rsyslog: rsyslogd startup succeeded
> Sep 25 22:44:08 server kernel: rklogd 1.19.7, log source = /proc/kmsg
> started.
> Sep 25 22:44:08 server kernel: rsyslogd[8110]: segfault at
> 000000000000000a
> rip 00000038c7568678 rsp 0000007fbfffdc2
> 0 error 4
> Sep 25 22:44:08 server rsyslog: rklogd startup succeeded
> Sep 25 22:44:08 server rsyslog: rsyslogd shutdown succeeded
>
> The above machine is a DL380 G4 with one CPU running hypthreaded. On
> another
> machine that's the same as that:
>
> Sep 25 22:42:24 server2 kernel: Kernel logging (proc) stopped.
> Sep 25 22:42:24 server2 kernel: Kernel log daemon terminating.
> Sep 25 22:42:25 server2 rsyslog: rklogd shutdown succeeded
> Sep 25 22:42:25 server2 rsyslogd: [origin software="rsyslogd"
> swVersion="1.19.7" x-pid="28497"] exiting on signal 15
> .
> Sep 25 22:43:43 server2 rsyslogd: [origin software="rsyslogd"
> swVersion="1.19.7" x-pid="16598"][x-configInfo udpRece
> ption="No" udpPort="514" tcpReception="No" tcpPort="0"] restart
> Sep 25 22:43:43 server2 rsyslog: rsyslogd startup succeeded
> Sep 25 22:43:43 server2 kernel: rklogd 1.19.7, log source = /proc/kmsg
> started.
> Sep 25 22:43:43 server2 kernel: rsyslogd[15998]: segfault at
> 000000000000000a
> rip 00000034d2868678 rsp 0000007fbfffd
> c20 error 4
> Sep 25 22:43:43 server2 rsyslog: rklogd startup succeeded
>
> Those two machines above are also running 64bit Linux.
>
> Using the same process on other 32 bit linux servers (DL360's), no
> segfaults
> occur.
>
> Note that I have never (ever) had rsyslog crash on me while it's
> running, and
> I currently run it on 6 servers to do the MySQL logging.
>
> Regards,
>
> Michael.
>
> > Rsyslog 1.19.7 is a code cleanup, bug fixing and maintenance
release.
> > Its primary goal is enhanced stability. Feature-wise, it offers
> support
> > of dropping NUL characters, which are often found at the end of
> > invalidly formatted syslog messages. Upgrading to this release is
> > recommended for all users.
> >
> > Changelog:
> >
> > http://www.rsyslog.com/Article129.phtml
> >
> > Download:
> >
> > http://www.rsyslog.com/Downloads-req-getit-lid-59.phtml
> >
> > PLEASE NOTE: the rsyslog site is currently migrated to a new
> > machine. In case you receive a temporary page, your DNS still caches
> > the old machine's address.
> >
> > As always, feedback is appreciated. Feedback is especially
> > appreciated from those that have experienced the segfault problem in
> > the past.
> >
> > Rainer Gerhards
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> ------- End of Original Message -------
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog