Mailing List Archive: gz omfile template corrupts file when used with multiple facilities

gz omfile template corrupts file when used with multiple facilities

May 25, 2022, 5:08 AM

Post #1 of 4 (358 views)

Hi

I hope I am not missing something obvious here.

I have a template which outputs logs to a gz file. I am using the template for three facilities (local 2, 5 and 6].

When I log a message using one of the facilities, let's say local2, the gz file is created as expected. However, if I use one of the other two facilities (right after i used local2), say local5, it will corrupt the gz file.

This happens if each action is declared on a different line inside config.

However, if I add the actions inside the configuration file on a single line, then it will not corrupt the file

In a nutshell:

The following template will corrupt log.gz file when using local2 AND either of the other two facilities.

$OMFileZipLevel 6
$template HOST_LOGS,"/var/log/%HOSTNAME%.log.gz"
local2.* -?HOST_LOGS
local5.* -?HOST_LOGS
local6.* -?HOST_LOGS

# logger -p local2.info local2
# zcat /var/log/test-srv.example.com.log.gz
2022-05-25T03:28:00.117395-07:00 test-srv.example.com lab: local2

# logger -p local2.info local2-2
# zcat /var/log/test-srv.example.com.log.gz
2022-05-25T03:28:00.117395-07:00 test-srv.example.com lab: local2
2022-05-25T03:29:20.113595-07:00 test-srv.example.com lab: local2-2

# logger -p local5.info local5
# zcat /var/log/test-srv.example.com.log.gz

gzip: test-srv.example.com.log.gz: invalid compressed data--format violated

The following template will NOT corrupt log.gz

$OMFileZipLevel 6
$template HOST_LOGS,"/var/log/%HOSTNAME%.log.gz"
local2.*, local5.*, local6.* -?HOST_LOGS

# logger -p local2.info local2
# zcat /vat/log/test-srv.example.com.log.gz
2022-05-25T03:29:51.937856-07:00 test-srv.example.com lab: local2

# logger -p local5.info local5
# zcat test-srv.example.com.log.gz
2022-05-25T03:29:51.937856-07:00 stest-srv.example.com lab: local2
2022-05-25T03:30:08.403518-07:00 test-srv.example.com lab: local5

# logger -p local6.info local6
# zcat test-srv.example.com.log.gz
2022-05-25T03:29:51.937856-07:00 test-srv.example.com lab: local2
2022-05-25T03:30:08.403518-07:00 test-srv.example.com lab: local5
2022-05-25T03:30:17.589634-07:00 test-srv.example.com lab: local6

# logger -p local2.info local2
# zcat test-srv.example.com.log.gz
2022-05-25T03:29:51.937856-07:00 test-srv.example.com lab: local2
2022-05-25T03:30:08.403518-07:00 test-srv.example.com lab: local5
2022-05-25T03:30:17.589634-07:00 test-srv.example.com lab: local6
2022-05-25T03:30:27.214990-07:00 test-srv.example.com lab: local2

Anyone has faced something similar before?
What could possibly cause this? Some sort of race condition opening the gz file for writing ?

Thanks
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: gz omfile template corrupts file when used with multiple facilities [ In reply to ]

rsyslog at lists

May 25, 2022, 9:11 AM

Post #2 of 4 (358 views)

Permalink

I try not to have multiple actions write to the same file, you can have the
multiple actions call a single ruleset that then writes to the file

i.e.

local2.* call write
local5.* call write
local6.* call write

ruleset(name="write"){
?HOST_LOGS
}

David Lang

On Wed, 25 May 2022, Dimi Onobodies via rsyslog wrote:

> Date: Wed, 25 May 2022 12:08:45 +0000
> From: Dimi Onobodies via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Dimi Onobodies <dimi_kdj@hotmail.com>
> Subject: [rsyslog] gz omfile template corrupts file when used with multiple
> facilities
>
> Hi
>
> I hope I am not missing something obvious here.
>
> I have a template which outputs logs to a gz file. I am using the template for three facilities (local 2, 5 and 6].
>
> When I log a message using one of the facilities, let's say local2, the gz file is created as expected. However, if I use one of the other two facilities (right after i used local2), say local5, it will corrupt the gz file.
>
> This happens if each action is declared on a different line inside config.
>
> However, if I add the actions inside the configuration file on a single line, then it will not corrupt the file
>
>
> In a nutshell:
>
> The following template will corrupt log.gz file when using local2 AND either of the other two facilities.
>
> $OMFileZipLevel 6
> $template HOST_LOGS,"/var/log/%HOSTNAME%.log.gz"
> local2.* -?HOST_LOGS
> local5.* -?HOST_LOGS
> local6.* -?HOST_LOGS
>
>
> # logger -p local2.info local2
> # zcat /var/log/test-srv.example.com.log.gz
> 2022-05-25T03:28:00.117395-07:00 test-srv.example.com lab: local2
>
> # logger -p local2.info local2-2
> # zcat /var/log/test-srv.example.com.log.gz
> 2022-05-25T03:28:00.117395-07:00 test-srv.example.com lab: local2
> 2022-05-25T03:29:20.113595-07:00 test-srv.example.com lab: local2-2
>
> # logger -p local5.info local5
> # zcat /var/log/test-srv.example.com.log.gz
>
> gzip: test-srv.example.com.log.gz: invalid compressed data--format violated
>
>
>
> The following template will NOT corrupt log.gz
>
> $OMFileZipLevel 6
> $template HOST_LOGS,"/var/log/%HOSTNAME%.log.gz"
> local2.*, local5.*, local6.* -?HOST_LOGS
>
>
>
> # logger -p local2.info local2
> # zcat /vat/log/test-srv.example.com.log.gz
> 2022-05-25T03:29:51.937856-07:00 test-srv.example.com lab: local2
>
>
> # logger -p local5.info local5
> # zcat test-srv.example.com.log.gz
> 2022-05-25T03:29:51.937856-07:00 stest-srv.example.com lab: local2
> 2022-05-25T03:30:08.403518-07:00 test-srv.example.com lab: local5
>
> # logger -p local6.info local6
> # zcat test-srv.example.com.log.gz
> 2022-05-25T03:29:51.937856-07:00 test-srv.example.com lab: local2
> 2022-05-25T03:30:08.403518-07:00 test-srv.example.com lab: local5
> 2022-05-25T03:30:17.589634-07:00 test-srv.example.com lab: local6
>
> # logger -p local2.info local2
> # zcat test-srv.example.com.log.gz
> 2022-05-25T03:29:51.937856-07:00 test-srv.example.com lab: local2
> 2022-05-25T03:30:08.403518-07:00 test-srv.example.com lab: local5
> 2022-05-25T03:30:17.589634-07:00 test-srv.example.com lab: local6
> 2022-05-25T03:30:27.214990-07:00 test-srv.example.com lab: local2
>
> Anyone has faced something similar before?
> What could possibly cause this? Some sort of race condition opening the gz file for writing ?
>
> Thanks
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: gz omfile template corrupts file when used with multiple facilities [ In reply to ]

rsyslog at lists

May 25, 2022, 9:40 AM

Post #3 of 4 (358 views)

Permalink

yes, that's the issue here.

one could also modify the filter to include all required local
facilities - but the call is more generic.

Rainer

El mié, 25 may 2022 a las 18:11, David Lang via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> I try not to have multiple actions write to the same file, you can have the
> multiple actions call a single ruleset that then writes to the file
>
> i.e.
>
> local2.* call write
> local5.* call write
> local6.* call write
>
> ruleset(name="write"){
> ?HOST_LOGS
> }
>
> David Lang
>
> On Wed, 25 May 2022, Dimi Onobodies via rsyslog wrote:
>
> > Date: Wed, 25 May 2022 12:08:45 +0000
> > From: Dimi Onobodies via rsyslog <rsyslog@lists.adiscon.com>
> > To: rsyslog-users <rsyslog@lists.adiscon.com>
> > Cc: Dimi Onobodies <dimi_kdj@hotmail.com>
> > Subject: [rsyslog] gz omfile template corrupts file when used with multiple
> > facilities
> >
> > Hi
> >
> > I hope I am not missing something obvious here.
> >
> > I have a template which outputs logs to a gz file. I am using the template for three facilities (local 2, 5 and 6].
> >
> > When I log a message using one of the facilities, let's say local2, the gz file is created as expected. However, if I use one of the other two facilities (right after i used local2), say local5, it will corrupt the gz file.
> >
> > This happens if each action is declared on a different line inside config.
> >
> > However, if I add the actions inside the configuration file on a single line, then it will not corrupt the file
> >
> >
> > In a nutshell:
> >
> > The following template will corrupt log.gz file when using local2 AND either of the other two facilities.
> >
> > $OMFileZipLevel 6
> > $template HOST_LOGS,"/var/log/%HOSTNAME%.log.gz"
> > local2.* -?HOST_LOGS
> > local5.* -?HOST_LOGS
> > local6.* -?HOST_LOGS
> >
> >
> > # logger -p local2.info local2
> > # zcat /var/log/test-srv.example.com.log.gz
> > 2022-05-25T03:28:00.117395-07:00 test-srv.example.com lab: local2
> >
> > # logger -p local2.info local2-2
> > # zcat /var/log/test-srv.example.com.log.gz
> > 2022-05-25T03:28:00.117395-07:00 test-srv.example.com lab: local2
> > 2022-05-25T03:29:20.113595-07:00 test-srv.example.com lab: local2-2
> >
> > # logger -p local5.info local5
> > # zcat /var/log/test-srv.example.com.log.gz
> >
> > gzip: test-srv.example.com.log.gz: invalid compressed data--format violated
> >
> >
> >
> > The following template will NOT corrupt log.gz
> >
> > $OMFileZipLevel 6
> > $template HOST_LOGS,"/var/log/%HOSTNAME%.log.gz"
> > local2.*, local5.*, local6.* -?HOST_LOGS
> >
> >
> >
> > # logger -p local2.info local2
> > # zcat /vat/log/test-srv.example.com.log.gz
> > 2022-05-25T03:29:51.937856-07:00 test-srv.example.com lab: local2
> >
> >
> > # logger -p local5.info local5
> > # zcat test-srv.example.com.log.gz
> > 2022-05-25T03:29:51.937856-07:00 stest-srv.example.com lab: local2
> > 2022-05-25T03:30:08.403518-07:00 test-srv.example.com lab: local5
> >
> > # logger -p local6.info local6
> > # zcat test-srv.example.com.log.gz
> > 2022-05-25T03:29:51.937856-07:00 test-srv.example.com lab: local2
> > 2022-05-25T03:30:08.403518-07:00 test-srv.example.com lab: local5
> > 2022-05-25T03:30:17.589634-07:00 test-srv.example.com lab: local6
> >
> > # logger -p local2.info local2
> > # zcat test-srv.example.com.log.gz
> > 2022-05-25T03:29:51.937856-07:00 test-srv.example.com lab: local2
> > 2022-05-25T03:30:08.403518-07:00 test-srv.example.com lab: local5
> > 2022-05-25T03:30:17.589634-07:00 test-srv.example.com lab: local6
> > 2022-05-25T03:30:27.214990-07:00 test-srv.example.com lab: local2
> >
> > Anyone has faced something similar before?
> > What could possibly cause this? Some sort of race condition opening the gz file for writing ?
> >
> > Thanks
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: gz omfile template corrupts file when used with multiple facilities [ In reply to ]

rsyslog at lists

May 26, 2022, 4:33 AM

Post #4 of 4 (354 views)

Permalink

Ok thanks for the help . Will apply your practices .
________________________________
From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Rainer Gerhards via rsyslog <rsyslog@lists.adiscon.com>
Sent: Wednesday, May 25, 2022 4:40 PM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>
Subject: Re: [rsyslog] gz omfile template corrupts file when used with multiple facilities

yes, that's the issue here.

one could also modify the filter to include all required local
facilities - but the call is more generic.

Rainer

El mi?, 25 may 2022 a las 18:11, David Lang via rsyslog
(<rsyslog@lists.adiscon.com>) escribi?:
>
> I try not to have multiple actions write to the same file, you can have the
> multiple actions call a single ruleset that then writes to the file
>
> i.e.
>
> local2.* call write
> local5.* call write
> local6.* call write
>
> ruleset(name="write"){
> ?HOST_LOGS
> }
>
> David Lang
>
> On Wed, 25 May 2022, Dimi Onobodies via rsyslog wrote:
>
> > Date: Wed, 25 May 2022 12:08:45 +0000
> > From: Dimi Onobodies via rsyslog <rsyslog@lists.adiscon.com>
> > To: rsyslog-users <rsyslog@lists.adiscon.com>
> > Cc: Dimi Onobodies <dimi_kdj@hotmail.com>
> > Subject: [rsyslog] gz omfile template corrupts file when used with multiple
> > facilities
> >
> > Hi
> >
> > I hope I am not missing something obvious here.
> >
> > I have a template which outputs logs to a gz file. I am using the template for three facilities (local 2, 5 and 6].
> >
> > When I log a message using one of the facilities, let's say local2, the gz file is created as expected. However, if I use one of the other two facilities (right after i used local2), say local5, it will corrupt the gz file.
> >
> > This happens if each action is declared on a different line inside config.
> >
> > However, if I add the actions inside the configuration file on a single line, then it will not corrupt the file
> >
> >
> > In a nutshell:
> >
> > The following template will corrupt log.gz file when using local2 AND either of the other two facilities.
> >
> > $OMFileZipLevel 6
> > $template HOST_LOGS,"/var/log/%HOSTNAME%.log.gz"
> > local2.* -?HOST_LOGS
> > local5.* -?HOST_LOGS
> > local6.* -?HOST_LOGS
> >
> >
> > # logger -p local2.info local2
> > # zcat /var/log/test-srv.example.com.log.gz
> > 2022-05-25T03:28:00.117395-07:00 test-srv.example.com lab: local2
> >
> > # logger -p local2.info local2-2
> > # zcat /var/log/test-srv.example.com.log.gz
> > 2022-05-25T03:28:00.117395-07:00 test-srv.example.com lab: local2
> > 2022-05-25T03:29:20.113595-07:00 test-srv.example.com lab: local2-2
> >
> > # logger -p local5.info local5
> > # zcat /var/log/test-srv.example.com.log.gz
> >
> > gzip: test-srv.example.com.log.gz: invalid compressed data--format violated
> >
> >
> >
> > The following template will NOT corrupt log.gz
> >
> > $OMFileZipLevel 6
> > $template HOST_LOGS,"/var/log/%HOSTNAME%.log.gz"
> > local2.*, local5.*, local6.* -?HOST_LOGS
> >
> >
> >
> > # logger -p local2.info local2
> > # zcat /vat/log/test-srv.example.com.log.gz
> > 2022-05-25T03:29:51.937856-07:00 test-srv.example.com lab: local2
> >
> >
> > # logger -p local5.info local5
> > # zcat test-srv.example.com.log.gz
> > 2022-05-25T03:29:51.937856-07:00 stest-srv.example.com lab: local2
> > 2022-05-25T03:30:08.403518-07:00 test-srv.example.com lab: local5
> >
> > # logger -p local6.info local6
> > # zcat test-srv.example.com.log.gz
> > 2022-05-25T03:29:51.937856-07:00 test-srv.example.com lab: local2
> > 2022-05-25T03:30:08.403518-07:00 test-srv.example.com lab: local5
> > 2022-05-25T03:30:17.589634-07:00 test-srv.example.com lab: local6
> >
> > # logger -p local2.info local2
> > # zcat test-srv.example.com.log.gz
> > 2022-05-25T03:29:51.937856-07:00 test-srv.example.com lab: local2
> > 2022-05-25T03:30:08.403518-07:00 test-srv.example.com lab: local5
> > 2022-05-25T03:30:17.589634-07:00 test-srv.example.com lab: local6
> > 2022-05-25T03:30:27.214990-07:00 test-srv.example.com lab: local2
> >
> > Anyone has faced something similar before?
> > What could possibly cause this? Some sort of race condition opening the gz file for writing ?
> >
> > Thanks
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.