Mailing List Archive

8.1905.0 Logging Stops
I have two EL6 hosts running rsyslog, one of which writes log files to disk while the other writes to a MySQL database.

I upgraded each host to 8.1905.0 Friday, 2019-05-30 and since that time, logging on both systems seems to stop for no apparent reason. I don't see any error messages.

Restarting rsyslog on each host restores functionality for 45-60 minutes, then it stops logging again. I have rolled back to 8.1904.0 and haven't seen this behavior repeat.

Is there a potential problem with 8.1905.0? The rsyslog configs on each EL host haven't changed since the upgrade Friday, nor have any other packages been installed/uninstalled.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
There does appear to be some problem with 8.1905, so far we don't have enough
information to figure out what. If it's possible to get a debug log and/or
pstats output from a box that's not logging it would help (ideally a debug log
that includes when it stops, but that's harder to get)

one bug we do know about is if the main queue and an action/ruleset queue are
configured for disk assist mode, rsyslog will segfault, but that doesn't match
what you are seeing.

We have at least one other report of 8.1905 stopping after a half hour or so.

David Lang

On Mon, 3 Jun 2019,
Heaton, Tobias via rsyslog wrote:

> Date: Mon, 3 Jun 2019 18:06:53 +0000
> From: "Heaton, Tobias via rsyslog" <rsyslog@lists.adiscon.com>
> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
> Cc: "Heaton, Tobias" <tobias.heaton@unh.edu>
> Subject: [rsyslog] 8.1905.0 Logging Stops
>
> I have two EL6 hosts running rsyslog, one of which writes log files to disk while the other writes to a MySQL database.
>
> I upgraded each host to 8.1905.0 Friday, 2019-05-30 and since that time, logging on both systems seems to stop for no apparent reason. I don't see any error messages.
>
> Restarting rsyslog on each host restores functionality for 45-60 minutes, then it stops logging again. I have rolled back to 8.1904.0 and haven't seen this behavior repeat.
>
> Is there a potential problem with 8.1905.0? The rsyslog configs on each EL host haven't changed since the upgrade Friday, nor have any other packages been installed/uninstalled.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
I'm trying to bump this thread - just tried upgrading again to 8.11905.0-2 and after about 10 minutes, all logging stops. No indication of any errors. Restarting rsyslog brings things back, but then 10 minutes later again, everything stops.

This is happening on two separate EL6 boxes. Rolling back to 8.1904.0-1 restores all functionality.

-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Heaton, Tobias via rsyslog
Sent: Monday, June 3, 2019 2:07 PM
To: rsyslog@lists.adiscon.com
Cc: Heaton, Tobias <tobias.heaton@unh.edu>
Subject: [rsyslog] 8.1905.0 Logging Stops

I have two EL6 hosts running rsyslog, one of which writes log files to disk while the other writes to a MySQL database.

I upgraded each host to 8.1905.0 Friday, 2019-05-30 and since that time, logging on both systems seems to stop for no apparent reason. I don't see any error messages.

Restarting rsyslog on each host restores functionality for 45-60 minutes, then it stops logging again. I have rolled back to 8.1904.0 and haven't seen this behavior repeat.

Is there a potential problem with 8.1905.0? The rsyslog configs on each EL host haven't changed since the upgrade Friday, nor have any other packages been installed/uninstalled.
_______________________________________________
rsyslog mailing list
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwICAg&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=69LNvD0eBSW_bd9n5hRj31IPXxnM93WteRS1aO85B5Y&s=2q-GoaOF6VeslY-8UgQExfBrLbo0ZaXisN9i5cDINbM&e=
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwICAg&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=69LNvD0eBSW_bd9n5hRj31IPXxnM93WteRS1aO85B5Y&s=U7eRJEj9cb1xwPtzTjtpJjVSEk0l54In0HgVbTdb_6M&e=
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
right now we have limited info on this problem

do you have impstats setup? can you show us what it outputs when logging stops?

if you can manage to get debug output over the time that it stops, that would be
wonderful.

David Lang

On Mon, 10 Jun 2019, Heaton, Tobias via rsyslog wrote:

> Date: Mon, 10 Jun 2019 14:28:50 +0000
> From: "Heaton, Tobias via rsyslog" <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: "Heaton, Tobias" <tobias.heaton@unh.edu>
> Subject: Re: [rsyslog] 8.1905.0 Logging Stops
>
> I'm trying to bump this thread - just tried upgrading again to 8.11905.0-2 and after about 10 minutes, all logging stops. No indication of any errors. Restarting rsyslog brings things back, but then 10 minutes later again, everything stops.
>
> This is happening on two separate EL6 boxes. Rolling back to 8.1904.0-1 restores all functionality.
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Heaton, Tobias via rsyslog
> Sent: Monday, June 3, 2019 2:07 PM
> To: rsyslog@lists.adiscon.com
> Cc: Heaton, Tobias <tobias.heaton@unh.edu>
> Subject: [rsyslog] 8.1905.0 Logging Stops
>
> I have two EL6 hosts running rsyslog, one of which writes log files to disk while the other writes to a MySQL database.
>
> I upgraded each host to 8.1905.0 Friday, 2019-05-30 and since that time, logging on both systems seems to stop for no apparent reason. I don't see any error messages.
>
> Restarting rsyslog on each host restores functionality for 45-60 minutes, then it stops logging again. I have rolled back to 8.1904.0 and haven't seen this behavior repeat.
>
> Is there a potential problem with 8.1905.0? The rsyslog configs on each EL host haven't changed since the upgrade Friday, nor have any other packages been installed/uninstalled.
> _______________________________________________
> rsyslog mailing list
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwICAg&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=69LNvD0eBSW_bd9n5hRj31IPXxnM93WteRS1aO85B5Y&s=2q-GoaOF6VeslY-8UgQExfBrLbo0ZaXisN9i5cDINbM&e=
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwICAg&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=69LNvD0eBSW_bd9n5hRj31IPXxnM93WteRS1aO85B5Y&s=U7eRJEj9cb1xwPtzTjtpJjVSEk0l54In0HgVbTdb_6M&e=
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
I configured impstats, updated rsyslog, restarted and awaited the failure. Tailing the busiest logs, behavior starts normal. Over the course of a few minutes, the logs seems to arrive in chunks...or at least appear to write that way to disk.

That behavior continues with logs populating incrementally slower and slower until it just stops. The impstats don't show anything of consequence - the repeated stats are all normal and the same both prior to and after the logs stop writing.

Rsyslog appears still running and impstats continue to populate again with no appreciable changes - very strange!


-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Monday, June 10, 2019 2:08 PM
To: Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
Cc: Heaton, Tobias <tobias.heaton@unh.edu>
Subject: Re: [rsyslog] 8.1905.0 Logging Stops

Caution - External Email;;;

********************************************************************************************************************************************
right now we have limited info on this problem



do you have impstats setup? can you show us what it outputs when logging stops?



if you can manage to get debug output over the time that it stops, that would be

wonderful.



David Lang



On Mon, 10 Jun 2019, Heaton, Tobias via rsyslog wrote:



> Date: Mon, 10 Jun 2019 14:28:50 +0000

> From: "Heaton, Tobias via rsyslog" <rsyslog@lists.adiscon.com>

> To: rsyslog-users <rsyslog@lists.adiscon.com>

> Cc: "Heaton, Tobias" <tobias.heaton@unh.edu>

> Subject: Re: [rsyslog] 8.1905.0 Logging Stops

>

> I'm trying to bump this thread - just tried upgrading again to 8.11905.0-2 and after about 10 minutes, all logging stops. No indication of any errors. Restarting rsyslog brings things back, but then 10 minutes later again, everything stops.

>

> This is happening on two separate EL6 boxes. Rolling back to 8.1904.0-1 restores all functionality.

>

> -----Original Message-----

> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Heaton, Tobias via rsyslog

> Sent: Monday, June 3, 2019 2:07 PM

> To: rsyslog@lists.adiscon.com

> Cc: Heaton, Tobias <tobias.heaton@unh.edu>

> Subject: [rsyslog] 8.1905.0 Logging Stops

>

> I have two EL6 hosts running rsyslog, one of which writes log files to disk while the other writes to a MySQL database.

>

> I upgraded each host to 8.1905.0 Friday, 2019-05-30 and since that time, logging on both systems seems to stop for no apparent reason. I don't see any error messages.

>

> Restarting rsyslog on each host restores functionality for 45-60 minutes, then it stops logging again. I have rolled back to 8.1904.0 and haven't seen this behavior repeat.

>

> Is there a potential problem with 8.1905.0? The rsyslog configs on each EL host haven't changed since the upgrade Friday, nor have any other packages been installed/uninstalled.

> _______________________________________________

> rsyslog mailing list

> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwICAg&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=69LNvD0eBSW_bd9n5hRj31IPXxnM93WteRS1aO85B5Y&s=2q-GoaOF6VeslY-8UgQExfBrLbo0ZaXisN9i5cDINbM&e=

> http://www.rsyslog.com/professional-services/

> What's up with rsyslog? Follow https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwICAg&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=69LNvD0eBSW_bd9n5hRj31IPXxnM93WteRS1aO85B5Y&s=U7eRJEj9cb1xwPtzTjtpJjVSEk0l54In0HgVbTdb_6M&e=

> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

> _______________________________________________

> rsyslog mailing list

> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIBAg&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=0w2XdtyZl5a52NEqzpJvoEWkZGPONmoDHL776Egzeuw&s=d-SZrocyZ9NIBE99melwdUvrw0yiTQKyli-e6xOcCy8&e=

> http://www.rsyslog.com/professional-services/

> What's up with rsyslog? Follow https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIBAg&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=0w2XdtyZl5a52NEqzpJvoEWkZGPONmoDHL776Egzeuw&s=FsdDeXNoDiy4qXbSpxVO7RTEdP8mRG-blTsbcPjCVTs&e=

> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

>

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
On Mon, 10 Jun 2019, Heaton, Tobias wrote:

> I configured impstats, updated rsyslog, restarted and awaited the failure. Tailing the busiest logs, behavior starts normal. Over the course of a few minutes, the logs seems to arrive in chunks...or at least appear to write that way to disk.
>
> That behavior continues with logs populating incrementally slower and slower until it just stops. The impstats don't show anything of consequence - the repeated stats are all normal and the same both prior to and after the logs stop writing.
>
> Rsyslog appears still running and impstats continue to populate again with no appreciable changes - very strange!

after it stops, does impstats show new messages arriving (and queues building)?
or does it show outputs failing and messages being discarded?
or something else?

a couple cycles of the pstats output after it's failed would be useful

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
Attached is a snippet of the impstats. Logs stopped writing at 15:00:01.

-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Monday, June 10, 2019 3:29 PM
To: Heaton, Tobias <tobias.heaton@unh.edu>
Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
Subject: RE: [rsyslog] 8.1905.0 Logging Stops

Caution - External Email;;;

********************************************************************************************************************************************
On Mon, 10 Jun 2019, Heaton, Tobias wrote:

> I configured impstats, updated rsyslog, restarted and awaited the failure. Tailing the busiest logs, behavior starts normal. Over the course of a few minutes, the logs seems to arrive in chunks...or at least appear to write that way to disk.
>
> That behavior continues with logs populating incrementally slower and slower until it just stops. The impstats don't show anything of consequence - the repeated stats are all normal and the same both prior to and after the logs stop writing.
>
> Rsyslog appears still running and impstats continue to populate again with no appreciable changes - very strange!

after it stops, does impstats show new messages arriving (and queues building)?
or does it show outputs failing and messages being discarded?
or something else?

a couple cycles of the pstats output after it's failed would be useful

David Lang
Re: 8.1905.0 Logging Stops [ In reply to ]
do you have impstats configured to reset it's counters each time it outputs?

David Lang

On Mon, 10 Jun 2019, Heaton, Tobias wrote:

> Date: Mon, 10 Jun 2019 19:37:31 +0000
> From: "Heaton, Tobias" <tobias.heaton@unh.edu>
> To: David Lang <david@lang.hm>
> Cc: "Heaton, Tobias via rsyslog" <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] 8.1905.0 Logging Stops
>
> Attached is a snippet of the impstats. Logs stopped writing at 15:00:01.
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Monday, June 10, 2019 3:29 PM
> To: Heaton, Tobias <tobias.heaton@unh.edu>
> Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] 8.1905.0 Logging Stops
>
> Caution - External Email;;;
>
> ********************************************************************************************************************************************
> On Mon, 10 Jun 2019, Heaton, Tobias wrote:
>
>> I configured impstats, updated rsyslog, restarted and awaited the failure. Tailing the busiest logs, behavior starts normal. Over the course of a few minutes, the logs seems to arrive in chunks...or at least appear to write that way to disk.
>>
>> That behavior continues with logs populating incrementally slower and slower until it just stops. The impstats don't show anything of consequence - the repeated stats are all normal and the same both prior to and after the logs stop writing.
>>
>> Rsyslog appears still running and impstats continue to populate again with no appreciable changes - very strange!
>
> after it stops, does impstats show new messages arriving (and queues building)?
> or does it show outputs failing and messages being discarded?
> or something else?
>
> a couple cycles of the pstats output after it's failed would be useful
>
> David Lang
>
Re: 8.1905.0 Logging Stops [ In reply to ]
I do - should I not?

-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Monday, June 10, 2019 3:43 PM
To: Heaton, Tobias <tobias.heaton@unh.edu>
Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
Subject: RE: [rsyslog] 8.1905.0 Logging Stops

Caution - External Email;;;

********************************************************************************************************************************************
do you have impstats configured to reset it's counters each time it outputs?

David Lang

On Mon, 10 Jun 2019, Heaton, Tobias wrote:

> Date: Mon, 10 Jun 2019 19:37:31 +0000
> From: "Heaton, Tobias" <tobias.heaton@unh.edu>
> To: David Lang <david@lang.hm>
> Cc: "Heaton, Tobias via rsyslog" <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] 8.1905.0 Logging Stops
>
> Attached is a snippet of the impstats. Logs stopped writing at 15:00:01.
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Monday, June 10, 2019 3:29 PM
> To: Heaton, Tobias <tobias.heaton@unh.edu>
> Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] 8.1905.0 Logging Stops
>
> Caution - External Email;;;
>
> ********************************************************************************************************************************************
> On Mon, 10 Jun 2019, Heaton, Tobias wrote:
>
>> I configured impstats, updated rsyslog, restarted and awaited the failure. Tailing the busiest logs, behavior starts normal. Over the course of a few minutes, the logs seems to arrive in chunks...or at least appear to write that way to disk.
>>
>> That behavior continues with logs populating incrementally slower and slower until it just stops. The impstats don't show anything of consequence - the repeated stats are all normal and the same both prior to and after the logs stop writing.
>>
>> Rsyslog appears still running and impstats continue to populate again with no appreciable changes - very strange!
>
> after it stops, does impstats show new messages arriving (and queues building)?
> or does it show outputs failing and messages being discarded?
> or something else?
>
> a couple cycles of the pstats output after it's failed would be useful
>
> David Lang
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
by the way, looking at your evicted stats, you probably want to tweak your
dynafilecachesize on some of your outputs (performance issue, probably not
related to this issue)

data was still being processed by outputs through the 15:00:17 window. It didn't
stop until the 15:00:22 window.

If you can get a debug log (send to Rainer) through the time it crashes, that
would be helpful

Also, can you post your config?

David Lang

On Mon, 10 Jun 2019, Heaton, Tobias wrote:

> Date: Mon, 10 Jun 2019 19:37:31 +0000
> From: "Heaton, Tobias" <tobias.heaton@unh.edu>
> To: David Lang <david@lang.hm>
> Cc: "Heaton, Tobias via rsyslog" <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] 8.1905.0 Logging Stops
>
> Attached is a snippet of the impstats. Logs stopped writing at 15:00:01.
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Monday, June 10, 2019 3:29 PM
> To: Heaton, Tobias <tobias.heaton@unh.edu>
> Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] 8.1905.0 Logging Stops
>
> Caution - External Email;;;
>
> ********************************************************************************************************************************************
> On Mon, 10 Jun 2019, Heaton, Tobias wrote:
>
>> I configured impstats, updated rsyslog, restarted and awaited the failure. Tailing the busiest logs, behavior starts normal. Over the course of a few minutes, the logs seems to arrive in chunks...or at least appear to write that way to disk.
>>
>> That behavior continues with logs populating incrementally slower and slower until it just stops. The impstats don't show anything of consequence - the repeated stats are all normal and the same both prior to and after the logs stop writing.
>>
>> Rsyslog appears still running and impstats continue to populate again with no appreciable changes - very strange!
>
> after it stops, does impstats show new messages arriving (and queues building)?
> or does it show outputs failing and messages being discarded?
> or something else?
>
> a couple cycles of the pstats output after it's failed would be useful
>
> David Lang
>
Re: 8.1905.0 Logging Stops [ In reply to ]
On Mon, 10 Jun 2019, Heaton, Tobias wrote:

> I do - should I not?

the stats are a little more inaccurate (there is a race condition between
the time that the stats are reported and when they are cleared that new events
can be lost) beyond that it's mostly preference, but the stats analysis tool on
the rsyslog.com website assumes that the counters are not reset.

I don't think it's a big deal, just confirming.

David Lang

> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Monday, June 10, 2019 3:43 PM
> To: Heaton, Tobias <tobias.heaton@unh.edu>
> Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] 8.1905.0 Logging Stops
>
> Caution - External Email;;;
>
> ********************************************************************************************************************************************
> do you have impstats configured to reset it's counters each time it outputs?
>
> David Lang
>
> On Mon, 10 Jun 2019, Heaton, Tobias wrote:
>
>> Date: Mon, 10 Jun 2019 19:37:31 +0000
>> From: "Heaton, Tobias" <tobias.heaton@unh.edu>
>> To: David Lang <david@lang.hm>
>> Cc: "Heaton, Tobias via rsyslog" <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] 8.1905.0 Logging Stops
>>
>> Attached is a snippet of the impstats. Logs stopped writing at 15:00:01.
>>
>> -----Original Message-----
>> From: David Lang <david@lang.hm>
>> Sent: Monday, June 10, 2019 3:29 PM
>> To: Heaton, Tobias <tobias.heaton@unh.edu>
>> Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] 8.1905.0 Logging Stops
>>
>> Caution - External Email;;;
>>
>> ********************************************************************************************************************************************
>> On Mon, 10 Jun 2019, Heaton, Tobias wrote:
>>
>>> I configured impstats, updated rsyslog, restarted and awaited the failure. Tailing the busiest logs, behavior starts normal. Over the course of a few minutes, the logs seems to arrive in chunks...or at least appear to write that way to disk.
>>>
>>> That behavior continues with logs populating incrementally slower and slower until it just stops. The impstats don't show anything of consequence - the repeated stats are all normal and the same both prior to and after the logs stop writing.
>>>
>>> Rsyslog appears still running and impstats continue to populate again with no appreciable changes - very strange!
>>
>> after it stops, does impstats show new messages arriving (and queues building)?
>> or does it show outputs failing and messages being discarded?
>> or something else?
>>
>> a couple cycles of the pstats output after it's failed would be useful
>>
>> David Lang
>>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
My apologies - I'm a little na?ve when it comes to the debugging.

What's your recommendation on the best way to capture debug?

-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Monday, June 10, 2019 3:50 PM
To: Heaton, Tobias <tobias.heaton@unh.edu>
Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
Subject: RE: [rsyslog] 8.1905.0 Logging Stops

Caution - External Email;;;

********************************************************************************************************************************************
by the way, looking at your evicted stats, you probably want to tweak your
dynafilecachesize on some of your outputs (performance issue, probably not
related to this issue)

data was still being processed by outputs through the 15:00:17 window. It didn't
stop until the 15:00:22 window.

If you can get a debug log (send to Rainer) through the time it crashes, that
would be helpful

Also, can you post your config?

David Lang

On Mon, 10 Jun 2019, Heaton, Tobias wrote:

> Date: Mon, 10 Jun 2019 19:37:31 +0000
> From: "Heaton, Tobias" <tobias.heaton@unh.edu>
> To: David Lang <david@lang.hm>
> Cc: "Heaton, Tobias via rsyslog" <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] 8.1905.0 Logging Stops
>
> Attached is a snippet of the impstats. Logs stopped writing at 15:00:01.
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Monday, June 10, 2019 3:29 PM
> To: Heaton, Tobias <tobias.heaton@unh.edu>
> Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] 8.1905.0 Logging Stops
>
> Caution - External Email;;;
>
> ********************************************************************************************************************************************
> On Mon, 10 Jun 2019, Heaton, Tobias wrote:
>
>> I configured impstats, updated rsyslog, restarted and awaited the failure. Tailing the busiest logs, behavior starts normal. Over the course of a few minutes, the logs seems to arrive in chunks...or at least appear to write that way to disk.
>>
>> That behavior continues with logs populating incrementally slower and slower until it just stops. The impstats don't show anything of consequence - the repeated stats are all normal and the same both prior to and after the logs stop writing.
>>
>> Rsyslog appears still running and impstats continue to populate again with no appreciable changes - very strange!
>
> after it stops, does impstats show new messages arriving (and queues building)?
> or does it show outputs failing and messages being discarded?
> or something else?
>
> a couple cycles of the pstats output after it's failed would be useful
>
> David Lang
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
Hi Tobias,

this is the "quick guide" to generate a debug log:
https://www.rsyslog.com/doc/v8-stable/troubleshooting/howtodebug.html
- just do the "regular debug" part and let it run until the problem
manifests.

We also have more elaborate ones, but I guess this one is sufficient
(let's keep it smooth and simple ;-)).

Rainer

El lun., 10 jun. 2019 a las 21:58, Heaton, Tobias via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> My apologies - I'm a little naïve when it comes to the debugging.
>
> What's your recommendation on the best way to capture debug?
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Monday, June 10, 2019 3:50 PM
> To: Heaton, Tobias <tobias.heaton@unh.edu>
> Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] 8.1905.0 Logging Stops
>
> Caution - External Email;;;
>
> ********************************************************************************************************************************************
> by the way, looking at your evicted stats, you probably want to tweak your
> dynafilecachesize on some of your outputs (performance issue, probably not
> related to this issue)
>
> data was still being processed by outputs through the 15:00:17 window. It didn't
> stop until the 15:00:22 window.
>
> If you can get a debug log (send to Rainer) through the time it crashes, that
> would be helpful
>
> Also, can you post your config?
>
> David Lang
>
> On Mon, 10 Jun 2019, Heaton, Tobias wrote:
>
> > Date: Mon, 10 Jun 2019 19:37:31 +0000
> > From: "Heaton, Tobias" <tobias.heaton@unh.edu>
> > To: David Lang <david@lang.hm>
> > Cc: "Heaton, Tobias via rsyslog" <rsyslog@lists.adiscon.com>
> > Subject: RE: [rsyslog] 8.1905.0 Logging Stops
> >
> > Attached is a snippet of the impstats. Logs stopped writing at 15:00:01.
> >
> > -----Original Message-----
> > From: David Lang <david@lang.hm>
> > Sent: Monday, June 10, 2019 3:29 PM
> > To: Heaton, Tobias <tobias.heaton@unh.edu>
> > Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
> > Subject: RE: [rsyslog] 8.1905.0 Logging Stops
> >
> > Caution - External Email;;;
> >
> > ********************************************************************************************************************************************
> > On Mon, 10 Jun 2019, Heaton, Tobias wrote:
> >
> >> I configured impstats, updated rsyslog, restarted and awaited the failure. Tailing the busiest logs, behavior starts normal. Over the course of a few minutes, the logs seems to arrive in chunks...or at least appear to write that way to disk.
> >>
> >> That behavior continues with logs populating incrementally slower and slower until it just stops. The impstats don't show anything of consequence - the repeated stats are all normal and the same both prior to and after the logs stop writing.
> >>
> >> Rsyslog appears still running and impstats continue to populate again with no appreciable changes - very strange!
> >
> > after it stops, does impstats show new messages arriving (and queues building)?
> > or does it show outputs failing and messages being discarded?
> > or something else?
> >
> > a couple cycles of the pstats output after it's failed would be useful
> >
> > David Lang
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
Hello Rainer,

Thank you for the information. I set the debug configuration, updated rsyslog and restarted. After about 5 minutes, the logs are starting to slow and the debug log is well over 12GB in size.

It seems having debug enabled slows general performance considerably and given the rate at which the debug log is building, finding relevant, helpful data might be a challenge...not to mention getting you the file itself!

Tobias

-----Original Message-----
From: Rainer Gerhards <rgerhards@hq.adiscon.com>
Sent: Tuesday, June 11, 2019 7:33 AM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: David Lang <david@lang.hm>; Heaton, Tobias <tobias.heaton@unh.edu>
Subject: Re: [rsyslog] 8.1905.0 Logging Stops

Caution - External Email;;;

********************************************************************************************************************************************
Hi Tobias,

this is the "quick guide" to generate a debug log:
https://www.rsyslog.com/doc/v8-stable/troubleshooting/howtodebug.html
- just do the "regular debug" part and let it run until the problem
manifests.

We also have more elaborate ones, but I guess this one is sufficient
(let's keep it smooth and simple ;-)).

Rainer

El lun., 10 jun. 2019 a las 21:58, Heaton, Tobias via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> My apologies - I'm a little naïve when it comes to the debugging.
>
> What's your recommendation on the best way to capture debug?
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Monday, June 10, 2019 3:50 PM
> To: Heaton, Tobias <tobias.heaton@unh.edu>
> Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] 8.1905.0 Logging Stops
>
> Caution - External Email;;;
>
> ********************************************************************************************************************************************
> by the way, looking at your evicted stats, you probably want to tweak your
> dynafilecachesize on some of your outputs (performance issue, probably not
> related to this issue)
>
> data was still being processed by outputs through the 15:00:17 window. It didn't
> stop until the 15:00:22 window.
>
> If you can get a debug log (send to Rainer) through the time it crashes, that
> would be helpful
>
> Also, can you post your config?
>
> David Lang
>
> On Mon, 10 Jun 2019, Heaton, Tobias wrote:
>
> > Date: Mon, 10 Jun 2019 19:37:31 +0000
> > From: "Heaton, Tobias" <tobias.heaton@unh.edu>
> > To: David Lang <david@lang.hm>
> > Cc: "Heaton, Tobias via rsyslog" <rsyslog@lists.adiscon.com>
> > Subject: RE: [rsyslog] 8.1905.0 Logging Stops
> >
> > Attached is a snippet of the impstats. Logs stopped writing at 15:00:01.
> >
> > -----Original Message-----
> > From: David Lang <david@lang.hm>
> > Sent: Monday, June 10, 2019 3:29 PM
> > To: Heaton, Tobias <tobias.heaton@unh.edu>
> > Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
> > Subject: RE: [rsyslog] 8.1905.0 Logging Stops
> >
> > Caution - External Email;;;
> >
> > ********************************************************************************************************************************************
> > On Mon, 10 Jun 2019, Heaton, Tobias wrote:
> >
> >> I configured impstats, updated rsyslog, restarted and awaited the failure. Tailing the busiest logs, behavior starts normal. Over the course of a few minutes, the logs seems to arrive in chunks...or at least appear to write that way to disk.
> >>
> >> That behavior continues with logs populating incrementally slower and slower until it just stops. The impstats don't show anything of consequence - the repeated stats are all normal and the same both prior to and after the logs stop writing.
> >>
> >> Rsyslog appears still running and impstats continue to populate again with no appreciable changes - very strange!
> >
> > after it stops, does impstats show new messages arriving (and queues building)?
> > or does it show outputs failing and messages being discarded?
> > or something else?
> >
> > a couple cycles of the pstats output after it's failed would be useful
> >
> > David Lang
> >
> _______________________________________________
> rsyslog mailing list
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=lhTjg_QDGIeHOeCAcLX5bXSSgEoxAn4zMhoKMn39_Q0&s=AIKzntUgGqWkTqo_j2lfE7vckKRK0v9bS913ZhCug6k&e=
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=lhTjg_QDGIeHOeCAcLX5bXSSgEoxAn4zMhoKMn39_Q0&s=Wv4TJP-48wkRkAh4kLGr1DRn9AJcn69V5upqPFFcXkY&e=
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
El mar., 11 jun. 2019 a las 14:33, Heaton, Tobias
(<tobias.heaton@unh.edu>) escribió:
>
> Hello Rainer,
>
> Thank you for the information. I set the debug configuration, updated rsyslog and restarted. After about 5 minutes, the logs are starting to slow and the debug log is well over 12GB in size.
>
> It seems having debug enabled slows general performance considerably and given the rate at which the debug log is building, finding relevant, helpful data might be a challenge...not to mention getting you the file itself!

If the 12GiB contain the actual error case, than it is not too bad - I
am used to wading though large debug files. These files usually
compress very well (xz recommended), so the transfer is also not too
bad. It would be best, though, if we can go to a point where the log
really stalls.

For starters, I would guess that the last minute or so of data could
be interesting.

Rainer
>
> Tobias
>
> -----Original Message-----
> From: Rainer Gerhards <rgerhards@hq.adiscon.com>
> Sent: Tuesday, June 11, 2019 7:33 AM
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: David Lang <david@lang.hm>; Heaton, Tobias <tobias.heaton@unh.edu>
> Subject: Re: [rsyslog] 8.1905.0 Logging Stops
>
> Caution - External Email;;;
>
> ********************************************************************************************************************************************
> Hi Tobias,
>
> this is the "quick guide" to generate a debug log:
> https://www.rsyslog.com/doc/v8-stable/troubleshooting/howtodebug.html
> - just do the "regular debug" part and let it run until the problem
> manifests.
>
> We also have more elaborate ones, but I guess this one is sufficient
> (let's keep it smooth and simple ;-)).
>
> Rainer
>
> El lun., 10 jun. 2019 a las 21:58, Heaton, Tobias via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
> >
> > My apologies - I'm a little naïve when it comes to the debugging.
> >
> > What's your recommendation on the best way to capture debug?
> >
> > -----Original Message-----
> > From: David Lang <david@lang.hm>
> > Sent: Monday, June 10, 2019 3:50 PM
> > To: Heaton, Tobias <tobias.heaton@unh.edu>
> > Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
> > Subject: RE: [rsyslog] 8.1905.0 Logging Stops
> >
> > Caution - External Email;;;
> >
> > ********************************************************************************************************************************************
> > by the way, looking at your evicted stats, you probably want to tweak your
> > dynafilecachesize on some of your outputs (performance issue, probably not
> > related to this issue)
> >
> > data was still being processed by outputs through the 15:00:17 window. It didn't
> > stop until the 15:00:22 window.
> >
> > If you can get a debug log (send to Rainer) through the time it crashes, that
> > would be helpful
> >
> > Also, can you post your config?
> >
> > David Lang
> >
> > On Mon, 10 Jun 2019, Heaton, Tobias wrote:
> >
> > > Date: Mon, 10 Jun 2019 19:37:31 +0000
> > > From: "Heaton, Tobias" <tobias.heaton@unh.edu>
> > > To: David Lang <david@lang.hm>
> > > Cc: "Heaton, Tobias via rsyslog" <rsyslog@lists.adiscon.com>
> > > Subject: RE: [rsyslog] 8.1905.0 Logging Stops
> > >
> > > Attached is a snippet of the impstats. Logs stopped writing at 15:00:01.
> > >
> > > -----Original Message-----
> > > From: David Lang <david@lang.hm>
> > > Sent: Monday, June 10, 2019 3:29 PM
> > > To: Heaton, Tobias <tobias.heaton@unh.edu>
> > > Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
> > > Subject: RE: [rsyslog] 8.1905.0 Logging Stops
> > >
> > > Caution - External Email;;;
> > >
> > > ********************************************************************************************************************************************
> > > On Mon, 10 Jun 2019, Heaton, Tobias wrote:
> > >
> > >> I configured impstats, updated rsyslog, restarted and awaited the failure. Tailing the busiest logs, behavior starts normal. Over the course of a few minutes, the logs seems to arrive in chunks...or at least appear to write that way to disk.
> > >>
> > >> That behavior continues with logs populating incrementally slower and slower until it just stops. The impstats don't show anything of consequence - the repeated stats are all normal and the same both prior to and after the logs stop writing.
> > >>
> > >> Rsyslog appears still running and impstats continue to populate again with no appreciable changes - very strange!
> > >
> > > after it stops, does impstats show new messages arriving (and queues building)?
> > > or does it show outputs failing and messages being discarded?
> > > or something else?
> > >
> > > a couple cycles of the pstats output after it's failed would be useful
> > >
> > > David Lang
> > >
> > _______________________________________________
> > rsyslog mailing list
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=lhTjg_QDGIeHOeCAcLX5bXSSgEoxAn4zMhoKMn39_Q0&s=AIKzntUgGqWkTqo_j2lfE7vckKRK0v9bS913ZhCug6k&e=
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=lhTjg_QDGIeHOeCAcLX5bXSSgEoxAn4zMhoKMn39_Q0&s=Wv4TJP-48wkRkAh4kLGr1DRn9AJcn69V5upqPFFcXkY&e=
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
Ok - I will set things up and do my best to capture the failure and let you know when I have logs.

Thanks for your help - appreciate it!

> On Jun 11, 2019, at 8:53 AM, Rainer Gerhards <rgerhards@hq.adiscon.com> wrote:
>
> Caution - External Email;;;
>
> ********************************************************************************************************************************************
> El mar., 11 jun. 2019 a las 14:33, Heaton, Tobias
> (<tobias.heaton@unh.edu>) escribió:
>>
>> Hello Rainer,
>>
>> Thank you for the information. I set the debug configuration, updated rsyslog and restarted. After about 5 minutes, the logs are starting to slow and the debug log is well over 12GB in size.
>>
>> It seems having debug enabled slows general performance considerably and given the rate at which the debug log is building, finding relevant, helpful data might be a challenge...not to mention getting you the file itself!
>
> If the 12GiB contain the actual error case, than it is not too bad - I
> am used to wading though large debug files. These files usually
> compress very well (xz recommended), so the transfer is also not too
> bad. It would be best, though, if we can go to a point where the log
> really stalls.
>
> For starters, I would guess that the last minute or so of data could
> be interesting.
>
> Rainer
>>
>> Tobias
>>
>> -----Original Message-----
>> From: Rainer Gerhards <rgerhards@hq.adiscon.com>
>> Sent: Tuesday, June 11, 2019 7:33 AM
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: David Lang <david@lang.hm>; Heaton, Tobias <tobias.heaton@unh.edu>
>> Subject: Re: [rsyslog] 8.1905.0 Logging Stops
>>
>> Caution - External Email;;;
>>
>> ********************************************************************************************************************************************
>> Hi Tobias,
>>
>> this is the "quick guide" to generate a debug log:
>> https://www.rsyslog.com/doc/v8-stable/troubleshooting/howtodebug.html
>> - just do the "regular debug" part and let it run until the problem
>> manifests.
>>
>> We also have more elaborate ones, but I guess this one is sufficient
>> (let's keep it smooth and simple ;-)).
>>
>> Rainer
>>
>> El lun., 10 jun. 2019 a las 21:58, Heaton, Tobias via rsyslog
>> (<rsyslog@lists.adiscon.com>) escribió:
>>>
>>> My apologies - I'm a little naïve when it comes to the debugging.
>>>
>>> What's your recommendation on the best way to capture debug?
>>>
>>> -----Original Message-----
>>> From: David Lang <david@lang.hm>
>>> Sent: Monday, June 10, 2019 3:50 PM
>>> To: Heaton, Tobias <tobias.heaton@unh.edu>
>>> Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
>>> Subject: RE: [rsyslog] 8.1905.0 Logging Stops
>>>
>>> Caution - External Email;;;
>>>
>>> ********************************************************************************************************************************************
>>> by the way, looking at your evicted stats, you probably want to tweak your
>>> dynafilecachesize on some of your outputs (performance issue, probably not
>>> related to this issue)
>>>
>>> data was still being processed by outputs through the 15:00:17 window. It didn't
>>> stop until the 15:00:22 window.
>>>
>>> If you can get a debug log (send to Rainer) through the time it crashes, that
>>> would be helpful
>>>
>>> Also, can you post your config?
>>>
>>> David Lang
>>>
>>>> On Mon, 10 Jun 2019, Heaton, Tobias wrote:
>>>>
>>>> Date: Mon, 10 Jun 2019 19:37:31 +0000
>>>> From: "Heaton, Tobias" <tobias.heaton@unh.edu>
>>>> To: David Lang <david@lang.hm>
>>>> Cc: "Heaton, Tobias via rsyslog" <rsyslog@lists.adiscon.com>
>>>> Subject: RE: [rsyslog] 8.1905.0 Logging Stops
>>>>
>>>> Attached is a snippet of the impstats. Logs stopped writing at 15:00:01.
>>>>
>>>> -----Original Message-----
>>>> From: David Lang <david@lang.hm>
>>>> Sent: Monday, June 10, 2019 3:29 PM
>>>> To: Heaton, Tobias <tobias.heaton@unh.edu>
>>>> Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
>>>> Subject: RE: [rsyslog] 8.1905.0 Logging Stops
>>>>
>>>> Caution - External Email;;;
>>>>
>>>> ********************************************************************************************************************************************
>>>>> On Mon, 10 Jun 2019, Heaton, Tobias wrote:
>>>>>
>>>>> I configured impstats, updated rsyslog, restarted and awaited the failure. Tailing the busiest logs, behavior starts normal. Over the course of a few minutes, the logs seems to arrive in chunks...or at least appear to write that way to disk.
>>>>>
>>>>> That behavior continues with logs populating incrementally slower and slower until it just stops. The impstats don't show anything of consequence - the repeated stats are all normal and the same both prior to and after the logs stop writing.
>>>>>
>>>>> Rsyslog appears still running and impstats continue to populate again with no appreciable changes - very strange!
>>>>
>>>> after it stops, does impstats show new messages arriving (and queues building)?
>>>> or does it show outputs failing and messages being discarded?
>>>> or something else?
>>>>
>>>> a couple cycles of the pstats output after it's failed would be useful
>>>>
>>>> David Lang
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=lhTjg_QDGIeHOeCAcLX5bXSSgEoxAn4zMhoKMn39_Q0&s=AIKzntUgGqWkTqo_j2lfE7vckKRK0v9bS913ZhCug6k&e=
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=lhTjg_QDGIeHOeCAcLX5bXSSgEoxAn4zMhoKMn39_Q0&s=Wv4TJP-48wkRkAh4kLGr1DRn9AJcn69V5upqPFFcXkY&e=
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
Rainer,

I traced my steps and allowed rsyslog to run for 45 minutes without seeing a failure.

The debug puts considerable strain on the system, so much so that the log timestamps started drifting. I was seeing logs stamped at 13:18 EST at 13:31EST actual time.

Furthermore the debug log is nearly 70GB, closing in on the available disk space.

Is there anything else I can try to help troubleshoot this problem?

Tobias

-----Original Message-----
From: Rainer Gerhards <rgerhards@hq.adiscon.com>
Sent: Tuesday, June 11, 2019 8:53 AM
To: Heaton, Tobias <tobias.heaton@unh.edu>
Cc: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang <david@lang.hm>
Subject: Re: [rsyslog] 8.1905.0 Logging Stops

Caution - External Email;;;

********************************************************************************************************************************************
El mar., 11 jun. 2019 a las 14:33, Heaton, Tobias
(<tobias.heaton@unh.edu>) escribió:
>
> Hello Rainer,
>
> Thank you for the information. I set the debug configuration, updated rsyslog and restarted. After about 5 minutes, the logs are starting to slow and the debug log is well over 12GB in size.
>
> It seems having debug enabled slows general performance considerably and given the rate at which the debug log is building, finding relevant, helpful data might be a challenge...not to mention getting you the file itself!

If the 12GiB contain the actual error case, than it is not too bad - I
am used to wading though large debug files. These files usually
compress very well (xz recommended), so the transfer is also not too
bad. It would be best, though, if we can go to a point where the log
really stalls.

For starters, I would guess that the last minute or so of data could
be interesting.

Rainer
>
> Tobias
>
> -----Original Message-----
> From: Rainer Gerhards <rgerhards@hq.adiscon.com>
> Sent: Tuesday, June 11, 2019 7:33 AM
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: David Lang <david@lang.hm>; Heaton, Tobias <tobias.heaton@unh.edu>
> Subject: Re: [rsyslog] 8.1905.0 Logging Stops
>
> Caution - External Email;;;
>
> ********************************************************************************************************************************************
> Hi Tobias,
>
> this is the "quick guide" to generate a debug log:
> https://www.rsyslog.com/doc/v8-stable/troubleshooting/howtodebug.html
> - just do the "regular debug" part and let it run until the problem
> manifests.
>
> We also have more elaborate ones, but I guess this one is sufficient
> (let's keep it smooth and simple ;-)).
>
> Rainer
>
> El lun., 10 jun. 2019 a las 21:58, Heaton, Tobias via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
> >
> > My apologies - I'm a little naïve when it comes to the debugging.
> >
> > What's your recommendation on the best way to capture debug?
> >
> > -----Original Message-----
> > From: David Lang <david@lang.hm>
> > Sent: Monday, June 10, 2019 3:50 PM
> > To: Heaton, Tobias <tobias.heaton@unh.edu>
> > Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
> > Subject: RE: [rsyslog] 8.1905.0 Logging Stops
> >
> > Caution - External Email;;;
> >
> > ********************************************************************************************************************************************
> > by the way, looking at your evicted stats, you probably want to tweak your
> > dynafilecachesize on some of your outputs (performance issue, probably not
> > related to this issue)
> >
> > data was still being processed by outputs through the 15:00:17 window. It didn't
> > stop until the 15:00:22 window.
> >
> > If you can get a debug log (send to Rainer) through the time it crashes, that
> > would be helpful
> >
> > Also, can you post your config?
> >
> > David Lang
> >
> > On Mon, 10 Jun 2019, Heaton, Tobias wrote:
> >
> > > Date: Mon, 10 Jun 2019 19:37:31 +0000
> > > From: "Heaton, Tobias" <tobias.heaton@unh.edu>
> > > To: David Lang <david@lang.hm>
> > > Cc: "Heaton, Tobias via rsyslog" <rsyslog@lists.adiscon.com>
> > > Subject: RE: [rsyslog] 8.1905.0 Logging Stops
> > >
> > > Attached is a snippet of the impstats. Logs stopped writing at 15:00:01.
> > >
> > > -----Original Message-----
> > > From: David Lang <david@lang.hm>
> > > Sent: Monday, June 10, 2019 3:29 PM
> > > To: Heaton, Tobias <tobias.heaton@unh.edu>
> > > Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
> > > Subject: RE: [rsyslog] 8.1905.0 Logging Stops
> > >
> > > Caution - External Email;;;
> > >
> > > ********************************************************************************************************************************************
> > > On Mon, 10 Jun 2019, Heaton, Tobias wrote:
> > >
> > >> I configured impstats, updated rsyslog, restarted and awaited the failure. Tailing the busiest logs, behavior starts normal. Over the course of a few minutes, the logs seems to arrive in chunks...or at least appear to write that way to disk.
> > >>
> > >> That behavior continues with logs populating incrementally slower and slower until it just stops. The impstats don't show anything of consequence - the repeated stats are all normal and the same both prior to and after the logs stop writing.
> > >>
> > >> Rsyslog appears still running and impstats continue to populate again with no appreciable changes - very strange!
> > >
> > > after it stops, does impstats show new messages arriving (and queues building)?
> > > or does it show outputs failing and messages being discarded?
> > > or something else?
> > >
> > > a couple cycles of the pstats output after it's failed would be useful
> > >
> > > David Lang
> > >
> > _______________________________________________
> > rsyslog mailing list
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=lhTjg_QDGIeHOeCAcLX5bXSSgEoxAn4zMhoKMn39_Q0&s=AIKzntUgGqWkTqo_j2lfE7vckKRK0v9bS913ZhCug6k&e=
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=lhTjg_QDGIeHOeCAcLX5bXSSgEoxAn4zMhoKMn39_Q0&s=Wv4TJP-48wkRkAh4kLGr1DRn9AJcn69V5upqPFFcXkY&e=
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
El mar., 11 jun. 2019 a las 19:23, Heaton, Tobias
(<tobias.heaton@unh.edu>) escribió:
>
> Rainer,
>
> I traced my steps and allowed rsyslog to run for 45 minutes without seeing a failure.
>
> The debug puts considerable strain on the system, so much so that the log timestamps started drifting. I was seeing logs stamped at 13:18 EST at 13:31EST actual time.
>
> Furthermore the debug log is nearly 70GB, closing in on the available disk space.
>
> Is there anything else I can try to help troubleshoot this problem?

Not really. Maybe the first 12 GiB actually contain info that get's us
a bit closer. So if you could share that, it would be useful. We can
fine-tune the debug output, but as we have not the slightest idea of
what may be the root cause that's no option.

Rainer
>
> Tobias
>
> -----Original Message-----
> From: Rainer Gerhards <rgerhards@hq.adiscon.com>
> Sent: Tuesday, June 11, 2019 8:53 AM
> To: Heaton, Tobias <tobias.heaton@unh.edu>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang <david@lang.hm>
> Subject: Re: [rsyslog] 8.1905.0 Logging Stops
>
> Caution - External Email;;;
>
> ********************************************************************************************************************************************
> El mar., 11 jun. 2019 a las 14:33, Heaton, Tobias
> (<tobias.heaton@unh.edu>) escribió:
> >
> > Hello Rainer,
> >
> > Thank you for the information. I set the debug configuration, updated rsyslog and restarted. After about 5 minutes, the logs are starting to slow and the debug log is well over 12GB in size.
> >
> > It seems having debug enabled slows general performance considerably and given the rate at which the debug log is building, finding relevant, helpful data might be a challenge...not to mention getting you the file itself!
>
> If the 12GiB contain the actual error case, than it is not too bad - I
> am used to wading though large debug files. These files usually
> compress very well (xz recommended), so the transfer is also not too
> bad. It would be best, though, if we can go to a point where the log
> really stalls.
>
> For starters, I would guess that the last minute or so of data could
> be interesting.
>
> Rainer
> >
> > Tobias
> >
> > -----Original Message-----
> > From: Rainer Gerhards <rgerhards@hq.adiscon.com>
> > Sent: Tuesday, June 11, 2019 7:33 AM
> > To: rsyslog-users <rsyslog@lists.adiscon.com>
> > Cc: David Lang <david@lang.hm>; Heaton, Tobias <tobias.heaton@unh.edu>
> > Subject: Re: [rsyslog] 8.1905.0 Logging Stops
> >
> > Caution - External Email;;;
> >
> > ********************************************************************************************************************************************
> > Hi Tobias,
> >
> > this is the "quick guide" to generate a debug log:
> > https://www.rsyslog.com/doc/v8-stable/troubleshooting/howtodebug.html
> > - just do the "regular debug" part and let it run until the problem
> > manifests.
> >
> > We also have more elaborate ones, but I guess this one is sufficient
> > (let's keep it smooth and simple ;-)).
> >
> > Rainer
> >
> > El lun., 10 jun. 2019 a las 21:58, Heaton, Tobias via rsyslog
> > (<rsyslog@lists.adiscon.com>) escribió:
> > >
> > > My apologies - I'm a little naïve when it comes to the debugging.
> > >
> > > What's your recommendation on the best way to capture debug?
> > >
> > > -----Original Message-----
> > > From: David Lang <david@lang.hm>
> > > Sent: Monday, June 10, 2019 3:50 PM
> > > To: Heaton, Tobias <tobias.heaton@unh.edu>
> > > Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
> > > Subject: RE: [rsyslog] 8.1905.0 Logging Stops
> > >
> > > Caution - External Email;;;
> > >
> > > ********************************************************************************************************************************************
> > > by the way, looking at your evicted stats, you probably want to tweak your
> > > dynafilecachesize on some of your outputs (performance issue, probably not
> > > related to this issue)
> > >
> > > data was still being processed by outputs through the 15:00:17 window. It didn't
> > > stop until the 15:00:22 window.
> > >
> > > If you can get a debug log (send to Rainer) through the time it crashes, that
> > > would be helpful
> > >
> > > Also, can you post your config?
> > >
> > > David Lang
> > >
> > > On Mon, 10 Jun 2019, Heaton, Tobias wrote:
> > >
> > > > Date: Mon, 10 Jun 2019 19:37:31 +0000
> > > > From: "Heaton, Tobias" <tobias.heaton@unh.edu>
> > > > To: David Lang <david@lang.hm>
> > > > Cc: "Heaton, Tobias via rsyslog" <rsyslog@lists.adiscon.com>
> > > > Subject: RE: [rsyslog] 8.1905.0 Logging Stops
> > > >
> > > > Attached is a snippet of the impstats. Logs stopped writing at 15:00:01.
> > > >
> > > > -----Original Message-----
> > > > From: David Lang <david@lang.hm>
> > > > Sent: Monday, June 10, 2019 3:29 PM
> > > > To: Heaton, Tobias <tobias.heaton@unh.edu>
> > > > Cc: David Lang <david@lang.hm>; Heaton, Tobias via rsyslog <rsyslog@lists.adiscon.com>
> > > > Subject: RE: [rsyslog] 8.1905.0 Logging Stops
> > > >
> > > > Caution - External Email;;;
> > > >
> > > > ********************************************************************************************************************************************
> > > > On Mon, 10 Jun 2019, Heaton, Tobias wrote:
> > > >
> > > >> I configured impstats, updated rsyslog, restarted and awaited the failure. Tailing the busiest logs, behavior starts normal. Over the course of a few minutes, the logs seems to arrive in chunks...or at least appear to write that way to disk.
> > > >>
> > > >> That behavior continues with logs populating incrementally slower and slower until it just stops. The impstats don't show anything of consequence - the repeated stats are all normal and the same both prior to and after the logs stop writing.
> > > >>
> > > >> Rsyslog appears still running and impstats continue to populate again with no appreciable changes - very strange!
> > > >
> > > > after it stops, does impstats show new messages arriving (and queues building)?
> > > > or does it show outputs failing and messages being discarded?
> > > > or something else?
> > > >
> > > > a couple cycles of the pstats output after it's failed would be useful
> > > >
> > > > David Lang
> > > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=lhTjg_QDGIeHOeCAcLX5bXSSgEoxAn4zMhoKMn39_Q0&s=AIKzntUgGqWkTqo_j2lfE7vckKRK0v9bS913ZhCug6k&e=
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=lhTjg_QDGIeHOeCAcLX5bXSSgEoxAn4zMhoKMn39_Q0&s=Wv4TJP-48wkRkAh4kLGr1DRn9AJcn69V5upqPFFcXkY&e=
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
Am Di., 11. Juni 2019 um 19:26 Uhr schrieb Rainer Gerhards via rsyslog
<rsyslog@lists.adiscon.com>:
>
> El mar., 11 jun. 2019 a las 19:23, Heaton, Tobias
> (<tobias.heaton@unh.edu>) escribió:
> >
> > Rainer,
> >
> > I traced my steps and allowed rsyslog to run for 45 minutes without seeing a failure.
> >
> > The debug puts considerable strain on the system, so much so that the log timestamps started drifting. I was seeing logs stamped at 13:18 EST at 13:31EST actual time.
> >
> > Furthermore the debug log is nearly 70GB, closing in on the available disk space.
> >
> > Is there anything else I can try to help troubleshoot this problem?
>
> Not really. Maybe the first 12 GiB actually contain info that get's us
> a bit closer. So if you could share that, it would be useful. We can
> fine-tune the debug output, but as we have not the slightest idea of
> what may be the root cause that's no option.
>

If the problem is reliably reproducible (which seems to be on affected
systems), maybe a git bisect would actually be the quickest way to
find the commit causing this.


--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
El mar., 11 jun. 2019 a las 20:16, Michael Biebl (<mbiebl@gmail.com>) escribió:
> If the problem is reliably reproducible (which seems to be on affected
> systems), maybe a git bisect would actually be the quickest way to
> find the commit causing this.

Good point!

@tobias: can you build from source?

If not, we could work with git bisect and custom packages, but that
looks very time-consuming.

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
FYI: I guess this thread here is a duplicate of
https://github.com/rsyslog/rsyslog/issues/3686

Knowing the full config would be useful. Can be created via

$ rsyslogd -f /etc/rsyslog.conf -o /tmp/full_rsyslog_conf

where -f should point to the main config file and the file pointed to
by -o contains the full config. Post the latter.

Rainer

El mié., 12 jun. 2019 a las 10:48, Rainer Gerhards
(<rgerhards@hq.adiscon.com>) escribió:
>
> El mar., 11 jun. 2019 a las 20:16, Michael Biebl (<mbiebl@gmail.com>) escribió:
> > If the problem is reliably reproducible (which seems to be on affected
> > systems), maybe a git bisect would actually be the quickest way to
> > find the commit causing this.
>
> Good point!
>
> @tobias: can you build from source?
>
> If not, we could work with git bisect and custom packages, but that
> looks very time-consuming.
>
> Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
Hello Rainer,

Am 12.06.2019 um 12:48 schrieb Rainer Gerhards via rsyslog:
> FYI: I guess this thread here is a duplicate of
> https://github.com/rsyslog/rsyslog/issues/3686
>
> Knowing the full config would be useful. Can be created via
>
> $ rsyslogd -f /etc/rsyslog.conf -o /tmp/full_rsyslog_conf
>
> where -f should point to the main config file and the file pointed to
> by -o contains the full config. Post the latter.

[...]

I tried this on
- openSUSE 4.12.14-lp150.12.61-default
rsyslogd 8.33.1
- Ubuntu 4.18.0-20-generic
rsyslogd 8.32.0
- Eisfair-64 3.16.68-eisfair-64-VIRT
rsyslogd 8.1904.0

The result is always

/usr/sbin/rsyslogd: invalid option -- 'o'
usage: rsyslogd [options]
use "man rsyslogd" for details. To run rsyslog interactively, use
"rsyslogd -n"
to run it in debug mode use "rsyslogd -dn"
For further information see https://www.rsyslog.com/doc/

Looks like a bug?

Thanks.
Ansgar
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
Looking at the aforementioned link, my config is considerably simpler. I did read about the suggested bump to the devel release, but given our syslog server is production I hesitate to continue testing and risk further loss of data.

> On Jun 12, 2019, at 6:49 AM, Rainer Gerhards <rgerhards@hq.adiscon.com> wrote:
>
> Caution - External Email;;;
>
> ********************************************************************************************************************************************
> FYI: I guess this thread here is a duplicate of
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rsyslog_rsyslog_issues_3686&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=S3Rb53yyV69wLOmt0FRCSHHGXxB8GD7vZAdnCB7qtYo&s=kstKSKXQNf__Rvz2xQVW5tWJfOyx-yNejVXfraa7SxY&e=
>
> Knowing the full config would be useful. Can be created via
>
> $ rsyslogd -f /etc/rsyslog.conf -o /tmp/full_rsyslog_conf
>
> where -f should point to the main config file and the file pointed to
> by -o contains the full config. Post the latter.
>
> Rainer
>
> El mié., 12 jun. 2019 a las 10:48, Rainer Gerhards
> (<rgerhards@hq.adiscon.com>) escribió:
>>
>>> El mar., 11 jun. 2019 a las 20:16, Michael Biebl (<mbiebl@gmail.com>) escribió:
>>> If the problem is reliably reproducible (which seems to be on affected
>>> systems), maybe a git bisect would actually be the quickest way to
>>> find the commit causing this.
>>
>> Good point!
>>
>> @tobias: can you build from source?
>>
>> If not, we could work with git bisect and custom packages, but that
>> looks very time-consuming.
>>
>> Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
Hi Ansgar,

glad to hear from you. What you have is simply a too-old version, the
-o switch is a recent addition of 8.1905.0 (2 weeks ago or so). The
version you have, I think, is around 9 month old.

Do you experience the same issue? That would be very interesting.

Rainer

El jue., 13 jun. 2019 a las 17:49, Ansgar Puester via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> Hello Rainer,
>
> Am 12.06.2019 um 12:48 schrieb Rainer Gerhards via rsyslog:
> > FYI: I guess this thread here is a duplicate of
> > https://github.com/rsyslog/rsyslog/issues/3686
> >
> > Knowing the full config would be useful. Can be created via
> >
> > $ rsyslogd -f /etc/rsyslog.conf -o /tmp/full_rsyslog_conf
> >
> > where -f should point to the main config file and the file pointed to
> > by -o contains the full config. Post the latter.
>
> [...]
>
> I tried this on
> - openSUSE 4.12.14-lp150.12.61-default
> rsyslogd 8.33.1
> - Ubuntu 4.18.0-20-generic
> rsyslogd 8.32.0
> - Eisfair-64 3.16.68-eisfair-64-VIRT
> rsyslogd 8.1904.0
>
> The result is always
>
> /usr/sbin/rsyslogd: invalid option -- 'o'
> usage: rsyslogd [options]
> use "man rsyslogd" for details. To run rsyslog interactively, use
> "rsyslogd -n"
> to run it in debug mode use "rsyslogd -dn"
> For further information see https://www.rsyslog.com/doc/
>
> Looks like a bug?
>
> Thanks.
> Ansgar
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: 8.1905.0 Logging Stops [ In reply to ]
El jue., 13 jun. 2019 a las 19:04, Heaton, Tobias
(<tobias.heaton@unh.edu>) escribió:
>
> Looking at the aforementioned link, my config is considerably simpler. I did read about the suggested bump to the devel release, but given our syslog server is production I hesitate to continue testing and risk further loss of data.

The daily stable build is actually a stable build, just updated more
frequently. It's called "devel" for historical reasons. Have a look
here:

https://www.rsyslog.com/downloads/download-daily-build/

Rainer
>
> > On Jun 12, 2019, at 6:49 AM, Rainer Gerhards <rgerhards@hq.adiscon.com> wrote:
> >
> > Caution - External Email;;;
> >
> > ********************************************************************************************************************************************
> > FYI: I guess this thread here is a duplicate of
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rsyslog_rsyslog_issues_3686&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=S3Rb53yyV69wLOmt0FRCSHHGXxB8GD7vZAdnCB7qtYo&s=kstKSKXQNf__Rvz2xQVW5tWJfOyx-yNejVXfraa7SxY&e=
> >
> > Knowing the full config would be useful. Can be created via
> >
> > $ rsyslogd -f /etc/rsyslog.conf -o /tmp/full_rsyslog_conf
> >
> > where -f should point to the main config file and the file pointed to
> > by -o contains the full config. Post the latter.
> >
> > Rainer
> >
> > El mié., 12 jun. 2019 a las 10:48, Rainer Gerhards
> > (<rgerhards@hq.adiscon.com>) escribió:
> >>
> >>> El mar., 11 jun. 2019 a las 20:16, Michael Biebl (<mbiebl@gmail.com>) escribió:
> >>> If the problem is reliably reproducible (which seems to be on affected
> >>> systems), maybe a git bisect would actually be the quickest way to
> >>> find the commit causing this.
> >>
> >> Good point!
> >>
> >> @tobias: can you build from source?
> >>
> >> If not, we could work with git bisect and custom packages, but that
> >> looks very time-consuming.
> >>
> >> Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

1 2  View All