Mailing List Archive

Altering forwarded logfile names
I have a configured central log collector using rsyslog. A few of the
devices forwarding their logs are appliances that have no configuration
options other than the IP forwarding address and protocol. I cannot control
what file names are being sent.

Unfortunately, they are sending unintelligible file names with characters
that normally would be escaped. Is there any way I can control or alter the
incoming file name to normalize it to avoid these odd characters?

For example, could I establish a character map that maps the unallowed
character to something acceptable?

thanks,

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.822*

*E*: Scott.Slattery@MotorolaSolutions.com

--


*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
Your use of the term “file name” is confusing. When senders deliver to rsyslog over the network there is no exchange of files or filenames, only packets of information. Those packets are expected to be in a format that syslog understands such that useful information (header elements and message body) may be parsed from them. If you as the rsyslog admin choose to use some of that header information to compose filenames for output files, then yes you are sort of at the mercy of the senders content (especially if the sender doesn’t follow the syslog rules). However, there are functions in the advanced syntax that can be used to perform the type of character replacements you’re talking about.

It is common practice to use the syslog header/rsyslog property element called “hostname” for just such purposes. Is this what you’re talking about? You’d have to provide your configuration for real analysis, at least the part you perceive to be responsible for the problem.

Regards,



> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <rsyslog@lists.adiscon.com> wrote:
>
> I have a configured central log collector using rsyslog. A few of the
> devices forwarding their logs are appliances that have no configuration
> options other than the IP forwarding address and protocol. I cannot control
> what file names are being sent.
>
> Unfortunately, they are sending unintelligible file names with characters
> that normally would be escaped. Is there any way I can control or alter the
> incoming file name to normalize it to avoid these odd characters?
>
> For example, could I establish a character map that maps the unallowed
> character to something acceptable?
>
> thanks,
>
> *Scott Slattery*
>
> *Sr. Systems & Cloud Architect*
>
> *Cloud, Compute, Information & Architecture Team*
>
> motorolasolutions.com
>
> *O: 602.529.822*
>
> *E*: Scott.Slattery@MotorolaSolutions.com
>
> --
>
>
> *For more information on how and why we collect your personal
> information, please visit our Privacy Policy
> <https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
can you show us a sample log?

Rsyslog does have template options that will restrict non-printable characters
(converting them to \xxx octal notation if you want)

but we would need to see your config to have any way to figure out what's going
on, and a sample of a bad log message (written with the template
RYSLOG_DebugFormat would help a huge amount)

David Lang

On Tue, 23 Mar 2021, John Chivian via rsyslog wrote:

> Date: Tue, 23 Mar 2021 17:30:14 -0500
> From: John Chivian via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: John Chivian <jchivian@chivian.com>
> Subject: Re: [rsyslog] Altering forwarded logfile names
>
> Your use of the term “file name” is confusing. When senders deliver to rsyslog over the network there is no exchange of files or filenames, only packets of information. Those packets are expected to be in a format that syslog understands such that useful information (header elements and message body) may be parsed from them. If you as the rsyslog admin choose to use some of that header information to compose filenames for output files, then yes you are sort of at the mercy of the senders content (especially if the sender doesn’t follow the syslog rules). However, there are functions in the advanced syntax that can be used to perform the type of character replacements you’re talking about.
>
> It is common practice to use the syslog header/rsyslog property element called “hostname” for just such purposes. Is this what you’re talking about? You’d have to provide your configuration for real analysis, at least the part you perceive to be responsible for the problem.
>
> Regards,
>
>
>
>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <rsyslog@lists.adiscon.com> wrote:
>>
>> I have a configured central log collector using rsyslog. A few of the
>> devices forwarding their logs are appliances that have no configuration
>> options other than the IP forwarding address and protocol. I cannot control
>> what file names are being sent.
>>
>> Unfortunately, they are sending unintelligible file names with characters
>> that normally would be escaped. Is there any way I can control or alter the
>> incoming file name to normalize it to avoid these odd characters?
>>
>> For example, could I establish a character map that maps the unallowed
>> character to something acceptable?
>>
>> thanks,
>>
>> *Scott Slattery*
>>
>> *Sr. Systems & Cloud Architect*
>>
>> *Cloud, Compute, Information & Architecture Team*
>>
>> motorolasolutions.com
>>
>> *O: 602.529.822*
>>
>> *E*: Scott.Slattery@MotorolaSolutions.com
>>
>> --
>>
>>
>> *For more information on how and why we collect your personal
>> information, please visit our Privacy Policy
>> <https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
Thanks, John, let me try to clarify what I mean.

Normally when I forward from a remote server to the central log server, I
can include a tag that can then be used to determine the file name I want
on the central server. Since I have no real way to include this tag from
the appliance, this is not an option.

I'm looking for a way of inspecting the incoming packets to determining the
source logfile name (which is included in the payload) and use that
filename on the target central server. Since there are multiple logs being
sent (access, audit, monitor, etc.), I'd like to segregate these into their
own files. I'm already using a template with the host information to
dynamically create the file names. I just don't know how I can go beyond
this to also include the source logname.

Here's the template I'm using. It works for all other hosts where I can
configure the tag but I get garbage names from the appliance. I had hoped
that the appliance included some standard syslog tags but it doesn't seem
so.

template(name="DynRemoteLogFile" type="string"
string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: Scott.Slattery@MotorolaSolutions.com




On Tue, Mar 23, 2021 at 3:30 PM John Chivian <jchivian@chivian.com> wrote:

> Your use of the term “file name” is confusing. When senders deliver to
> rsyslog over the network there is no exchange of files or filenames, only
> packets of information. Those packets are expected to be in a format that
> syslog understands such that useful information (header elements and
> message body) may be parsed from them. If you as the rsyslog admin choose
> to use some of that header information to compose filenames for output
> files, then yes you are sort of at the mercy of the senders content
> (especially if the sender doesn’t follow the syslog rules). However, there
> are functions in the advanced syntax that can be used to perform the type
> of character replacements you’re talking about.
>
> It is common practice to use the syslog header/rsyslog property element
> called “hostname” for just such purposes. Is this what you’re talking
> about? You’d have to provide your configuration for real analysis, at
> least the part you perceive to be responsible for the problem.
>
> Regards,
>
>
>
> > On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
> rsyslog@lists.adiscon.com> wrote:
> >
> > I have a configured central log collector using rsyslog. A few of the
> > devices forwarding their logs are appliances that have no configuration
> > options other than the IP forwarding address and protocol. I cannot
> control
> > what file names are being sent.
> >
> > Unfortunately, they are sending unintelligible file names with characters
> > that normally would be escaped. Is there any way I can control or alter
> the
> > incoming file name to normalize it to avoid these odd characters?
> >
> > For example, could I establish a character map that maps the unallowed
> > character to something acceptable?
> >
> > thanks,
> >
> > *Scott Slattery*
> >
> > *Sr. Systems & Cloud Architect*
> >
> > *Cloud, Compute, Information & Architecture Team*
> >
> > motorolasolutions.com
> >
> > *O: 602.529.822*
> >
> > *E*: Scott.Slattery@MotorolaSolutions.com
> >
> > --
> >
> >
> > *For more information on how and why we collect your personal
> > information, please visit our Privacy Policy
> > <
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >.*
> > _______________________________________________
> > rsyslog mailing list
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
> > What's up with rsyslog? Follow
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
>

--


*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
the source logfile name is not included in the payload by the syslog spec. It
may be in the case of your appliance, but we would need to see a sample log to
understand ho to parse it.

based on your template, you are using app-name, which may be listed separtely if
it's a RFC5424 format log, or may be part of the syslog tag if it's a RFC3164
format log over the wire (neither format has a way to specify a source log file
by default)

you can look at https://github.com/Chojins/LinuxCNC-Polargraph and see the *-cc
options that you could apply to the app-name to eliminate control characters.

Again, we really need to see the original log message to understand what's what.
Please log it with the templateRSYSLOG_DebugFormat so we can see exactly what is
sent over the wire and how rsyslog has parsed it.

David Lang

On Tue, 23 Mar 2021, Scott Slattery via rsyslog
wrote:

> Date: Tue, 23 Mar 2021 16:05:45 -0700
> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
> To: John Chivian <jchivian@chivian.com>
> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
> rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] Altering forwarded logfile names
>
> Thanks, John, let me try to clarify what I mean.
>
> Normally when I forward from a remote server to the central log server, I
> can include a tag that can then be used to determine the file name I want
> on the central server. Since I have no real way to include this tag from
> the appliance, this is not an option.
>
> I'm looking for a way of inspecting the incoming packets to determining the
> source logfile name (which is included in the payload) and use that
> filename on the target central server. Since there are multiple logs being
> sent (access, audit, monitor, etc.), I'd like to segregate these into their
> own files. I'm already using a template with the host information to
> dynamically create the file names. I just don't know how I can go beyond
> this to also include the source logname.
>
> Here's the template I'm using. It works for all other hosts where I can
> configure the tag but I get garbage names from the appliance. I had hoped
> that the appliance included some standard syslog tags but it doesn't seem
> so.
>
> template(name="DynRemoteLogFile" type="string"
> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
>
> *Scott Slattery*
>
> *Sr. Systems & Cloud Architect*
>
> *Cloud, Compute, Information & Architecture Team*
>
> motorolasolutions.com
>
> *O: 602.529.8226*
>
> *E*: Scott.Slattery@MotorolaSolutions.com
>
>
>
>
> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <jchivian@chivian.com> wrote:
>
>> Your use of the term “file name” is confusing. When senders deliver to
>> rsyslog over the network there is no exchange of files or filenames, only
>> packets of information. Those packets are expected to be in a format that
>> syslog understands such that useful information (header elements and
>> message body) may be parsed from them. If you as the rsyslog admin choose
>> to use some of that header information to compose filenames for output
>> files, then yes you are sort of at the mercy of the senders content
>> (especially if the sender doesn’t follow the syslog rules). However, there
>> are functions in the advanced syntax that can be used to perform the type
>> of character replacements you’re talking about.
>>
>> It is common practice to use the syslog header/rsyslog property element
>> called “hostname” for just such purposes. Is this what you’re talking
>> about? You’d have to provide your configuration for real analysis, at
>> least the part you perceive to be responsible for the problem.
>>
>> Regards,
>>
>>
>>
>> > On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
>> rsyslog@lists.adiscon.com> wrote:
>> >
>> > I have a configured central log collector using rsyslog. A few of the
>> > devices forwarding their logs are appliances that have no configuration
>> > options other than the IP forwarding address and protocol. I cannot
>> control
>> > what file names are being sent.
>> >
>> > Unfortunately, they are sending unintelligible file names with characters
>> > that normally would be escaped. Is there any way I can control or alter
>> the
>> > incoming file name to normalize it to avoid these odd characters?
>> >
>> > For example, could I establish a character map that maps the unallowed
>> > character to something acceptable?
>> >
>> > thanks,
>> >
>> > *Scott Slattery*
>> >
>> > *Sr. Systems & Cloud Architect*
>> >
>> > *Cloud, Compute, Information & Architecture Team*
>> >
>> > motorolasolutions.com
>> >
>> > *O: 602.529.822*
>> >
>> > *E*: Scott.Slattery@MotorolaSolutions.com
>> >
>> > --
>> >
>> >
>> > *For more information on how and why we collect your personal
>> > information, please visit our Privacy Policy
>> > <
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>> >.*
>> > _______________________________________________
>> > rsyslog mailing list
>> >
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
>> >
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
>> > What's up with rsyslog? Follow
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>>
>
> --
>
>
> *For more information on how and why we collect your personal
> information, please visit our Privacy Policy
> <https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
Hi David, fortunately I had already done this. I'm including an actual log
entry but have anonymized the data to keep the actual user and email
address confidential:

Debug line with all properties:
FROMHOST: 'ause1oagatst02.aws.mycompany.com'
, fromhost-ip: '10.41.102.143'
, HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
, PRI: 13,
syslogtag '03-23T16:'
, programname: '03-23T16'
, APP-NAME: '03-23T16'
, PROCID: '-'
, MSGID: '-'
, TIMNESTAMP: 'Mar 23 21:47:13'
, STRUCTURED-DATA: '-'
, *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY
ACCESS AUTHZ SESSION INFO USER_SESSION
[.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
Joe.User@mycompantions.com" APP="Ignio Uat OAG"
APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com"
RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
10.44.65.38#0122021-03-23T16:47:20.708-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
UserName=Joe.User@mycompantions.com
SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompan'
*escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
[.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
Joe.User@mycompantions.com" APP="Ignio Uat OAG"
APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com"
RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
10.44.65.38#0122021-03-23T16:47:20.708-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
UserName=Joe.User@mycompantions.com
SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompan'
*rawmsg*: '03-23T16:47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
[.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
Joe.User@mycompantions.com" APP="Ignio Uat OAG"
APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com"
RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
10.44.65.38#0122021-03-23T16:47:20.708-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
UserName=Joe.User@mycompantions.com
SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompan'


By inspecting the rawmsg, I can see that field four (space-delimited)
indicates this is the ACCESS log. So if I were able to extract the log
identifier from the msg, I could then write all access logs to the same
daily file. There are other formats as well from the same device but the
idea is the same.

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: Scott.Slattery@MotorolaSolutions.com




On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm> wrote:

> the source logfile name is not included in the payload by the syslog spec.
> It
> may be in the case of your appliance, but we would need to see a sample
> log to
> understand ho to parse it.
>
> based on your template, you are using app-name, which may be listed
> separtely if
> it's a RFC5424 format log, or may be part of the syslog tag if it's a
> RFC3164
> format log over the wire (neither format has a way to specify a source log
> file
> by default)
>
> you can look at
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
> and see the *-cc
> options that you could apply to the app-name to eliminate control
> characters.
>
> Again, we really need to see the original log message to understand what's
> what.
> Please log it with the templateRSYSLOG_DebugFormat so we can see exactly
> what is
> sent over the wire and how rsyslog has parsed it.
>
> David Lang
>
> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
> wrote:
>
> > Date: Tue, 23 Mar 2021 16:05:45 -0700
> > From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
> > To: John Chivian <jchivian@chivian.com>
> > Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
> > rsyslog-users <rsyslog@lists.adiscon.com>
> > Subject: Re: [rsyslog] Altering forwarded logfile names
> >
> > Thanks, John, let me try to clarify what I mean.
> >
> > Normally when I forward from a remote server to the central log server, I
> > can include a tag that can then be used to determine the file name I want
> > on the central server. Since I have no real way to include this tag from
> > the appliance, this is not an option.
> >
> > I'm looking for a way of inspecting the incoming packets to determining
> the
> > source logfile name (which is included in the payload) and use that
> > filename on the target central server. Since there are multiple logs
> being
> > sent (access, audit, monitor, etc.), I'd like to segregate these into
> their
> > own files. I'm already using a template with the host information to
> > dynamically create the file names. I just don't know how I can go beyond
> > this to also include the source logname.
> >
> > Here's the template I'm using. It works for all other hosts where I can
> > configure the tag but I get garbage names from the appliance. I had hoped
> > that the appliance included some standard syslog tags but it doesn't seem
> > so.
> >
> > template(name="DynRemoteLogFile" type="string"
> >
> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
> >
> > *Scott Slattery*
> >
> > *Sr. Systems & Cloud Architect*
> >
> > *Cloud, Compute, Information & Architecture Team*
> >
> > motorolasolutions.com
> >
> > *O: 602.529.8226*
> >
> > *E*: Scott.Slattery@MotorolaSolutions.com
> >
> >
> >
> >
> > On Tue, Mar 23, 2021 at 3:30 PM John Chivian <jchivian@chivian.com>
> wrote:
> >
> >> Your use of the term “file name” is confusing. When senders deliver to
> >> rsyslog over the network there is no exchange of files or filenames,
> only
> >> packets of information. Those packets are expected to be in a format
> that
> >> syslog understands such that useful information (header elements and
> >> message body) may be parsed from them. If you as the rsyslog admin
> choose
> >> to use some of that header information to compose filenames for output
> >> files, then yes you are sort of at the mercy of the senders content
> >> (especially if the sender doesn’t follow the syslog rules). However,
> there
> >> are functions in the advanced syntax that can be used to perform the
> type
> >> of character replacements you’re talking about.
> >>
> >> It is common practice to use the syslog header/rsyslog property element
> >> called “hostname” for just such purposes. Is this what you’re talking
> >> about? You’d have to provide your configuration for real analysis, at
> >> least the part you perceive to be responsible for the problem.
> >>
> >> Regards,
> >>
> >>
> >>
> >> > On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
> >> rsyslog@lists.adiscon.com> wrote:
> >> >
> >> > I have a configured central log collector using rsyslog. A few of the
> >> > devices forwarding their logs are appliances that have no
> configuration
> >> > options other than the IP forwarding address and protocol. I cannot
> >> control
> >> > what file names are being sent.
> >> >
> >> > Unfortunately, they are sending unintelligible file names with
> characters
> >> > that normally would be escaped. Is there any way I can control or
> alter
> >> the
> >> > incoming file name to normalize it to avoid these odd characters?
> >> >
> >> > For example, could I establish a character map that maps the unallowed
> >> > character to something acceptable?
> >> >
> >> > thanks,
> >> >
> >> > *Scott Slattery*
> >> >
> >> > *Sr. Systems & Cloud Architect*
> >> >
> >> > *Cloud, Compute, Information & Architecture Team*
> >> >
> >> > motorolasolutions.com
> >> >
> >> > *O: 602.529.822*
> >> >
> >> > *E*: Scott.Slattery@MotorolaSolutions.com
> >> >
> >> > --
> >> >
> >> >
> >> > *For more information on how and why we collect your personal
> >> > information, please visit our Privacy Policy
> >> > <
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >> >.*
> >> > _______________________________________________
> >> > rsyslog mailing list
> >> >
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
> >> >
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
> >> > What's up with rsyslog? Follow
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> DON'T LIKE THAT.
> >>
> >>
> >
> > --
> >
> >
> > *For more information on how and why we collect your personal
> > information, please visit our Privacy Policy
> > <
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >.*
> > _______________________________________________
> > rsyslog mailing list
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
> > What's up with rsyslog? Follow
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.

--


*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
Ok, the problem here is that it's sending a completly invalid timestamp, which
is why it's showing up with the syslogtag/app-name as the beginning of the
timestamp.

I would start off by filing a ticket with the appliance vendor complaining that
they are not following the syslog RFCs (either the old or the new)

then what you will need to do is to have a conditional (probably based on
fromhost-ip) to identify these logs and run them through a different parser
(look at mmnormalize for a fast, flexible one, but you may be able to get away
with the field() function if you only need to do one extraction)

and then to output the logs, you will need a new template that uses the
variables that your parser pulls out of them since you can't use the default
properties.

David Lang

On Tue,
23 Mar 2021, Scott Slattery wrote:

> Date: Tue, 23 Mar 2021 17:51:28 -0700
> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> To: David Lang <david@lang.hm>
> Cc: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>,
> John Chivian <jchivian@chivian.com>
> Subject: Re: [rsyslog] Altering forwarded logfile names
>
> Hi David, fortunately I had already done this. I'm including an actual log
> entry but have anonymized the data to keep the actual user and email
> address confidential:
>
> Debug line with all properties:
> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
> , fromhost-ip: '10.41.102.143'
> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
> , PRI: 13,
> syslogtag '03-23T16:'
> , programname: '03-23T16'
> , APP-NAME: '03-23T16'
> , PROCID: '-'
> , MSGID: '-'
> , TIMNESTAMP: 'Mar 23 21:47:13'
> , STRUCTURED-DATA: '-'
> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY
> ACCESS AUTHZ SESSION INFO USER_SESSION
> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com"
> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
> UserName=Joe.User@mycompantions.com
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="Joe.User@mycompan'
> *escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com"
> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
> UserName=Joe.User@mycompantions.com
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="Joe.User@mycompan'
> *rawmsg*: '03-23T16:47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com"
> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
> UserName=Joe.User@mycompantions.com
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="Joe.User@mycompan'
>
>
> By inspecting the rawmsg, I can see that field four (space-delimited)
> indicates this is the ACCESS log. So if I were able to extract the log
> identifier from the msg, I could then write all access logs to the same
> daily file. There are other formats as well from the same device but the
> idea is the same.
>
> *Scott Slattery*
>
> *Sr. Systems & Cloud Architect*
>
> *Cloud, Compute, Information & Architecture Team*
>
> motorolasolutions.com
>
> *O: 602.529.8226*
>
> *E*: Scott.Slattery@MotorolaSolutions.com
>
>
>
>
> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm> wrote:
>
>> the source logfile name is not included in the payload by the syslog spec.
>> It
>> may be in the case of your appliance, but we would need to see a sample
>> log to
>> understand ho to parse it.
>>
>> based on your template, you are using app-name, which may be listed
>> separtely if
>> it's a RFC5424 format log, or may be part of the syslog tag if it's a
>> RFC3164
>> format log over the wire (neither format has a way to specify a source log
>> file
>> by default)
>>
>> you can look at
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
>> and see the *-cc
>> options that you could apply to the app-name to eliminate control
>> characters.
>>
>> Again, we really need to see the original log message to understand what's
>> what.
>> Please log it with the templateRSYSLOG_DebugFormat so we can see exactly
>> what is
>> sent over the wire and how rsyslog has parsed it.
>>
>> David Lang
>>
>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
>> wrote:
>>
>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
>>> To: John Chivian <jchivian@chivian.com>
>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
>>> rsyslog-users <rsyslog@lists.adiscon.com>
>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>
>>> Thanks, John, let me try to clarify what I mean.
>>>
>>> Normally when I forward from a remote server to the central log server, I
>>> can include a tag that can then be used to determine the file name I want
>>> on the central server. Since I have no real way to include this tag from
>>> the appliance, this is not an option.
>>>
>>> I'm looking for a way of inspecting the incoming packets to determining
>> the
>>> source logfile name (which is included in the payload) and use that
>>> filename on the target central server. Since there are multiple logs
>> being
>>> sent (access, audit, monitor, etc.), I'd like to segregate these into
>> their
>>> own files. I'm already using a template with the host information to
>>> dynamically create the file names. I just don't know how I can go beyond
>>> this to also include the source logname.
>>>
>>> Here's the template I'm using. It works for all other hosts where I can
>>> configure the tag but I get garbage names from the appliance. I had hoped
>>> that the appliance included some standard syslog tags but it doesn't seem
>>> so.
>>>
>>> template(name="DynRemoteLogFile" type="string"
>>>
>> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
>>>
>>> *Scott Slattery*
>>>
>>> *Sr. Systems & Cloud Architect*
>>>
>>> *Cloud, Compute, Information & Architecture Team*
>>>
>>> motorolasolutions.com
>>>
>>> *O: 602.529.8226*
>>>
>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>
>>>
>>>
>>>
>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <jchivian@chivian.com>
>> wrote:
>>>
>>>> Your use of the term “file name” is confusing. When senders deliver to
>>>> rsyslog over the network there is no exchange of files or filenames,
>> only
>>>> packets of information. Those packets are expected to be in a format
>> that
>>>> syslog understands such that useful information (header elements and
>>>> message body) may be parsed from them. If you as the rsyslog admin
>> choose
>>>> to use some of that header information to compose filenames for output
>>>> files, then yes you are sort of at the mercy of the senders content
>>>> (especially if the sender doesn’t follow the syslog rules). However,
>> there
>>>> are functions in the advanced syntax that can be used to perform the
>> type
>>>> of character replacements you’re talking about.
>>>>
>>>> It is common practice to use the syslog header/rsyslog property element
>>>> called “hostname” for just such purposes. Is this what you’re talking
>>>> about? You’d have to provide your configuration for real analysis, at
>>>> least the part you perceive to be responsible for the problem.
>>>>
>>>> Regards,
>>>>
>>>>
>>>>
>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
>>>> rsyslog@lists.adiscon.com> wrote:
>>>>>
>>>>> I have a configured central log collector using rsyslog. A few of the
>>>>> devices forwarding their logs are appliances that have no
>> configuration
>>>>> options other than the IP forwarding address and protocol. I cannot
>>>> control
>>>>> what file names are being sent.
>>>>>
>>>>> Unfortunately, they are sending unintelligible file names with
>> characters
>>>>> that normally would be escaped. Is there any way I can control or
>> alter
>>>> the
>>>>> incoming file name to normalize it to avoid these odd characters?
>>>>>
>>>>> For example, could I establish a character map that maps the unallowed
>>>>> character to something acceptable?
>>>>>
>>>>> thanks,
>>>>>
>>>>> *Scott Slattery*
>>>>>
>>>>> *Sr. Systems & Cloud Architect*
>>>>>
>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>
>>>>> motorolasolutions.com
>>>>>
>>>>> *O: 602.529.822*
>>>>>
>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>> *For more information on how and why we collect your personal
>>>>> information, please visit our Privacy Policy
>>>>> <
>>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>> .*
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
>>>>> What's up with rsyslog? Follow
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>>
>>>>
>>>
>>> --
>>>
>>>
>>> *For more information on how and why we collect your personal
>>> information, please visit our Privacy Policy
>>> <
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>> .*
>>> _______________________________________________
>>> rsyslog mailing list
>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
>>> What's up with rsyslog? Follow
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
David is right on. The “2021-“ is missing from the “03-23T16:47:20.708-05:00”



> On Mar 23, 2021, at 21:14, David Lang <david@lang.hm> wrote:
>
> ?Ok, the problem here is that it's sending a completly invalid timestamp, which is why it's showing up with the syslogtag/app-name as the beginning of the timestamp.
>
> I would start off by filing a ticket with the appliance vendor complaining that they are not following the syslog RFCs (either the old or the new)
>
> then what you will need to do is to have a conditional (probably based on fromhost-ip) to identify these logs and run them through a different parser (look at mmnormalize for a fast, flexible one, but you may be able to get away with the field() function if you only need to do one extraction)
>
> and then to output the logs, you will need a new template that uses the variables that your parser pulls out of them since you can't use the default properties.
>
> David Lang
>
>> On Tue, 23 Mar 2021, Scott Slattery wrote:
>>
>> Date: Tue, 23 Mar 2021 17:51:28 -0700
>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>> To: David Lang <david@lang.hm>
>> Cc: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>,
>> John Chivian <jchivian@chivian.com>
>> Subject: Re: [rsyslog] Altering forwarded logfile names
>> Hi David, fortunately I had already done this. I'm including an actual log
>> entry but have anonymized the data to keep the actual user and email
>> address confidential:
>>
>> Debug line with all properties:
>> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
>> , fromhost-ip: '10.41.102.143'
>> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
>> , PRI: 13,
>> syslogtag '03-23T16:'
>> , programname: '03-23T16'
>> , APP-NAME: '03-23T16'
>> , PROCID: '-'
>> , MSGID: '-'
>> , TIMNESTAMP: 'Mar 23 21:47:13'
>> , STRUCTURED-DATA: '-'
>> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY
>> ACCESS AUTHZ SESSION INFO USER_SESSION
>> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com"
>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
>> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
>> Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
>> UserName=Joe.User@mycompantions.com
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>> Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
>> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> SUBJECT="Joe.User@mycompan'
>> *escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
>> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
>> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com"
>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
>> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
>> Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
>> UserName=Joe.User@mycompantions.com
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>> Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
>> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> SUBJECT="Joe.User@mycompan'
>> *rawmsg*: '03-23T16:47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
>> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
>> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com"
>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
>> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
>> Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
>> UserName=Joe.User@mycompantions.com
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>> Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
>> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> SUBJECT="Joe.User@mycompan'
>>
>>
>> By inspecting the rawmsg, I can see that field four (space-delimited)
>> indicates this is the ACCESS log. So if I were able to extract the log
>> identifier from the msg, I could then write all access logs to the same
>> daily file. There are other formats as well from the same device but the
>> idea is the same.
>>
>> *Scott Slattery*
>>
>> *Sr. Systems & Cloud Architect*
>>
>> *Cloud, Compute, Information & Architecture Team*
>>
>> motorolasolutions.com
>>
>> *O: 602.529.8226*
>>
>> *E*: Scott.Slattery@MotorolaSolutions.com
>>
>>
>>
>>
>>> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm> wrote:
>>>
>>> the source logfile name is not included in the payload by the syslog spec.
>>> It
>>> may be in the case of your appliance, but we would need to see a sample
>>> log to
>>> understand ho to parse it.
>>>
>>> based on your template, you are using app-name, which may be listed
>>> separtely if
>>> it's a RFC5424 format log, or may be part of the syslog tag if it's a
>>> RFC3164
>>> format log over the wire (neither format has a way to specify a source log
>>> file
>>> by default)
>>>
>>> you can look at
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
>>> and see the *-cc
>>> options that you could apply to the app-name to eliminate control
>>> characters.
>>>
>>> Again, we really need to see the original log message to understand what's
>>> what.
>>> Please log it with the templateRSYSLOG_DebugFormat so we can see exactly
>>> what is
>>> sent over the wire and how rsyslog has parsed it.
>>>
>>> David Lang
>>>
>>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
>>> wrote:
>>>
>>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
>>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
>>>> To: John Chivian <jchivian@chivian.com>
>>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
>>>> rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>>
>>>> Thanks, John, let me try to clarify what I mean.
>>>>
>>>> Normally when I forward from a remote server to the central log server, I
>>>> can include a tag that can then be used to determine the file name I want
>>>> on the central server. Since I have no real way to include this tag from
>>>> the appliance, this is not an option.
>>>>
>>>> I'm looking for a way of inspecting the incoming packets to determining
>>> the
>>>> source logfile name (which is included in the payload) and use that
>>>> filename on the target central server. Since there are multiple logs
>>> being
>>>> sent (access, audit, monitor, etc.), I'd like to segregate these into
>>> their
>>>> own files. I'm already using a template with the host information to
>>>> dynamically create the file names. I just don't know how I can go beyond
>>>> this to also include the source logname.
>>>>
>>>> Here's the template I'm using. It works for all other hosts where I can
>>>> configure the tag but I get garbage names from the appliance. I had hoped
>>>> that the appliance included some standard syslog tags but it doesn't seem
>>>> so.
>>>>
>>>> template(name="DynRemoteLogFile" type="string"
>>>>
>>> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
>>>>
>>>> *Scott Slattery*
>>>>
>>>> *Sr. Systems & Cloud Architect*
>>>>
>>>> *Cloud, Compute, Information & Architecture Team*
>>>>
>>>> motorolasolutions.com
>>>>
>>>> *O: 602.529.8226*
>>>>
>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <jchivian@chivian.com>
>>> wrote:
>>>>
>>>>> Your use of the term “file name” is confusing. When senders deliver to
>>>>> rsyslog over the network there is no exchange of files or filenames,
>>> only
>>>>> packets of information. Those packets are expected to be in a format
>>> that
>>>>> syslog understands such that useful information (header elements and
>>>>> message body) may be parsed from them. If you as the rsyslog admin
>>> choose
>>>>> to use some of that header information to compose filenames for output
>>>>> files, then yes you are sort of at the mercy of the senders content
>>>>> (especially if the sender doesn’t follow the syslog rules). However,
>>> there
>>>>> are functions in the advanced syntax that can be used to perform the
>>> type
>>>>> of character replacements you’re talking about.
>>>>>
>>>>> It is common practice to use the syslog header/rsyslog property element
>>>>> called “hostname” for just such purposes. Is this what you’re talking
>>>>> about? You’d have to provide your configuration for real analysis, at
>>>>> least the part you perceive to be responsible for the problem.
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>>
>>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
>>>>> rsyslog@lists.adiscon.com> wrote:
>>>>>>
>>>>>> I have a configured central log collector using rsyslog. A few of the
>>>>>> devices forwarding their logs are appliances that have no
>>> configuration
>>>>>> options other than the IP forwarding address and protocol. I cannot
>>>>> control
>>>>>> what file names are being sent.
>>>>>>
>>>>>> Unfortunately, they are sending unintelligible file names with
>>> characters
>>>>>> that normally would be escaped. Is there any way I can control or
>>> alter
>>>>> the
>>>>>> incoming file name to normalize it to avoid these odd characters?
>>>>>>
>>>>>> For example, could I establish a character map that maps the unallowed
>>>>>> character to something acceptable?
>>>>>>
>>>>>> thanks,
>>>>>>
>>>>>> *Scott Slattery*
>>>>>>
>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>
>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>
>>>>>> motorolasolutions.com
>>>>>>
>>>>>> *O: 602.529.822*
>>>>>>
>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>
>>>>>> --
>>>>>>
>>>>>>
>>>>>> *For more information on how and why we collect your personal
>>>>>> information, please visit our Privacy Policy
>>>>>> <
>>>>>
>>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>> .*
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>>
>>>>>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
>>>>>>
>>>>>
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
>>>>>> What's up with rsyslog? Follow
>>>>>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>>
>>>> *For more information on how and why we collect your personal
>>>> information, please visit our Privacy Policy
>>>> <
>>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>> .*
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
>>>>
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
>>>> What's up with rsyslog? Follow
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Altering forwarded logfile names [ In reply to ]
There's something strange with this log entry. Notice that it looks as
if it were more than one log entries joined with LF character (escaped
on input as #012). And all "further" entries start with a - still
nonconformant to the RFC but "fuller" - timestamp containing year.
It's the leading event that seems to be trimmed in the front somehow. Do
all events arrive like this?
It looks as if the input wasn't properly splitting the stream into
single entries. Did you touch the frame delimiter setting on input?



Wys?ano z telefonu Samsung



-------- Original message --------
From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
Date: 24/03/2021 01:52 (GMT+01:00)
To: David Lang <david@lang.hm>
Cc: Scott Slattery <scott.slattery@motorolasolutions.com>, Scott
Slattery via rsyslog <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Altering forwarded logfile names


Hi David, fortunately I had already done this. I'm including an actual log
entry but have anonymized the data to keep the actual user and email
address confidential:

Debug line with all properties:
FROMHOST: 'ause1oagatst02.aws.mycompany.com'
, fromhost-ip: '10.41.102.143'
, HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
, PRI: 13,
syslogtag '03-23T16:'
, programname: '03-23T16'
, APP-NAME: '03-23T16'
, PROCID: '-'
, MSGID: '-'
, TIMNESTAMP: 'Mar 23 21:47:13'
, STRUCTURED-DATA: '-'
, *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY
ACCESS AUTHZ SESSION INFO USER_SESSION
[.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
Joe.User@mycompantions.com" APP="Ignio Uat OAG"
APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com"
RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
10.44.65.38#0122021-03-23T16:47:20.708-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
UserName=Joe.User@mycompantions.com
SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompan'
*escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
[.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
Joe.User@mycompantions.com" APP="Ignio Uat OAG"
APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com"
RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
10.44.65.38#0122021-03-23T16:47:20.708-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
UserName=Joe.User@mycompantions.com
SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompan'
*rawmsg*: '03-23T16:47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
[.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
Joe.User@mycompantions.com" APP="Ignio Uat OAG"
APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com"
RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
10.44.65.38#0122021-03-23T16:47:20.708-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
UserName=Joe.User@mycompantions.com
SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompan'


By inspecting the rawmsg, I can see that field four (space-delimited)
indicates this is the ACCESS log. So if I were able to extract the log
identifier from the msg, I could then write all access logs to the same
daily file. There are other formats as well from the same device but the
idea is the same.

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: Scott.Slattery@MotorolaSolutions.com




On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm> wrote:

> the source logfile name is not included in the payload by the syslog
spec.
> It
> may be in the case of your appliance, but we would need to see a sample
> log to
> understand ho to parse it.
>
> based on your template, you are using app-name, which may be listed
> separtely if
> it's a RFC5424 format log, or may be part of the syslog tag if it's a
> RFC3164
> format log over the wire (neither format has a way to specify a
source log
> file
> by default)
>
> you can look at
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
> and see the *-cc
> options that you could apply to the app-name to eliminate control
> characters.
>
> Again, we really need to see the original log message to understand
what's
> what.
> Please log it with the templateRSYSLOG_DebugFormat so we can see exactly
> what is
> sent over the wire and how rsyslog has parsed it.
>
> David Lang
>
> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
> wrote:
>
> > Date: Tue, 23 Mar 2021 16:05:45 -0700
> > From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
> > To: John Chivian <jchivian@chivian.com>
> > Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
> > rsyslog-users <rsyslog@lists.adiscon.com>
> > Subject: Re: [rsyslog] Altering forwarded logfile names
> >
> > Thanks, John, let me try to clarify what I mean.
> >
> > Normally when I forward from a remote server to the central log
server, I
> > can include a tag that can then be used to determine the file name
I want
> > on the central server. Since I have no real way to include this tag
from
> > the appliance, this is not an option.
> >
> > I'm looking for a way of inspecting the incoming packets to determining
> the
> > source logfile name (which is included in the payload) and use that
> > filename on the target central server. Since there are multiple logs
> being
> > sent (access, audit, monitor, etc.), I'd like to segregate these into
> their
> > own files. I'm already using a template with the host information to
> > dynamically create the file names. I just don't know how I can go
beyond
> > this to also include the source logname.
> >
> > Here's the template I'm using. It works for all other hosts where I can
> > configure the tag but I get garbage names from the appliance. I had
hoped
> > that the appliance included some standard syslog tags but it
doesn't seem
> > so.
> >
> > template(name="DynRemoteLogFile" type="string"
> >
>
string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
> >
> > *Scott Slattery*
> >
> > *Sr. Systems & Cloud Architect*
> >
> > *Cloud, Compute, Information & Architecture Team*
> >
> > motorolasolutions.com
> >
> > *O: 602.529.8226*
> >
> > *E*: Scott.Slattery@MotorolaSolutions.com
> >
> >
> >
> >
> > On Tue, Mar 23, 2021 at 3:30 PM John Chivian <jchivian@chivian.com>
> wrote:
> >
> >> Your use of the term “file name” is confusing. When senders deliver to
> >> rsyslog over the network there is no exchange of files or filenames,
> only
> >> packets of information. Those packets are expected to be in a format
> that
> >> syslog understands such that useful information (header elements and
> >> message body) may be parsed from them. If you as the rsyslog admin
> choose
> >> to use some of that header information to compose filenames for output
> >> files, then yes you are sort of at the mercy of the senders content
> >> (especially if the sender doesn’t follow the syslog rules). However,
> there
> >> are functions in the advanced syntax that can be used to perform the
> type
> >> of character replacements you’re talking about.
> >>
> >> It is common practice to use the syslog header/rsyslog property
element
> >> called “hostname” for just such purposes. Is this what you’re talking
> >> about? You’d have to provide your configuration for real analysis, at
> >> least the part you perceive to be responsible for the problem.
> >>
> >> Regards,
> >>
> >>
> >>
> >> > On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
> >> rsyslog@lists.adiscon.com> wrote:
> >> >
> >> > I have a configured central log collector using rsyslog. A few
of the
> >> > devices forwarding their logs are appliances that have no
> configuration
> >> > options other than the IP forwarding address and protocol. I cannot
> >> control
> >> > what file names are being sent.
> >> >
> >> > Unfortunately, they are sending unintelligible file names with
> characters
> >> > that normally would be escaped. Is there any way I can control or
> alter
> >> the
> >> > incoming file name to normalize it to avoid these odd characters?
> >> >
> >> > For example, could I establish a character map that maps the
unallowed
> >> > character to something acceptable?
> >> >
> >> > thanks,
> >> >
> >> > *Scott Slattery*
> >> >
> >> > *Sr. Systems & Cloud Architect*
> >> >
> >> > *Cloud, Compute, Information & Architecture Team*
> >> >
> >> > motorolasolutions.com
> >> >
> >> > *O: 602.529.822*
> >> >
> >> > *E*: Scott.Slattery@MotorolaSolutions.com
> >> >
> >> > --
> >> >
> >> >
> >> > *For more information on how and why we collect your personal
> >> > information, please visit our Privacy Policy
> >> > <
> >>
>
https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >> >.*
> >> > _______________________________________________
> >> > rsyslog mailing list
> >> >
> >>
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
> >> >
> >>
>
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
> >> > What's up with rsyslog? Follow
> >>
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> DON'T LIKE THAT.
> >>
> >>
> >
> > --
> >
> >
> > *For more information on how and why we collect your personal
> > information, please visit our Privacy Policy
> > <
>
https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >.*
> > _______________________________________________
> > rsyslog mailing list
> >
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
> >
>
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
> > What's up with rsyslog? Follow
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.

--


*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
if they are sending logs separated by lf instead of newline, then when they hit
maxmessagesize bytes, it will truncate one log and start the next log in the
middle of the message (which in this particular message may luck out to only
truncate the year)

look at the rawmsg field for several messages, do the starts all look the same
or are they all at different points in the message.

@mariusz, good catch, I just looked at the start of the message, I didn't wade
my way though the entire thing to notice the merged logs.

David Lang


On Wed, 24 Mar 2021, Mariusz Kruk via rsyslog wrote:

> Date: Wed, 24 Mar 2021 08:11:49 +0100
> From: Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Mariusz Kruk <mkr@safecomp.com>
> Subject: [rsyslog] Altering forwarded logfile names
>
> There's something strange with this log entry. Notice that it looks as
> if it were more than one log entries joined with LF character (escaped
> on input as #012). And all "further" entries start with a - still
> nonconformant to the RFC but "fuller" - timestamp containing year.
> It's the leading event that seems to be trimmed in the front somehow. Do
> all events arrive like this?
> It looks as if the input wasn't properly splitting the stream into
> single entries. Did you touch the frame delimiter setting on input?
>
>
>
> Wys?ano z telefonu Samsung
>
>
>
> -------- Original message --------
> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
> Date: 24/03/2021 01:52 (GMT+01:00)
> To: David Lang <david@lang.hm>
> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>, Scott
> Slattery via rsyslog <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] Altering forwarded logfile names
>
>
> Hi David, fortunately I had already done this. I'm including an actual log
> entry but have anonymized the data to keep the actual user and email
> address confidential:
>
> Debug line with all properties:
> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
> , fromhost-ip: '10.41.102.143'
> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
> , PRI: 13,
> syslogtag '03-23T16:'
> , programname: '03-23T16'
> , APP-NAME: '03-23T16'
> , PROCID: '-'
> , MSGID: '-'
> , TIMNESTAMP: 'Mar 23 21:47:13'
> , STRUCTURED-DATA: '-'
> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY
> ACCESS AUTHZ SESSION INFO USER_SESSION
> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com"
> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
> UserName=Joe.User@mycompantions.com
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="Joe.User@mycompan'
> *escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com"
> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
> UserName=Joe.User@mycompantions.com
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="Joe.User@mycompan'
> *rawmsg*: '03-23T16:47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com"
> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
> UserName=Joe.User@mycompantions.com
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="Joe.User@mycompan'
>
>
> By inspecting the rawmsg, I can see that field four (space-delimited)
> indicates this is the ACCESS log. So if I were able to extract the log
> identifier from the msg, I could then write all access logs to the same
> daily file. There are other formats as well from the same device but the
> idea is the same.
>
> *Scott Slattery*
>
> *Sr. Systems & Cloud Architect*
>
> *Cloud, Compute, Information & Architecture Team*
>
> motorolasolutions.com
>
> *O: 602.529.8226*
>
> *E*: Scott.Slattery@MotorolaSolutions.com
>
>
>
>
> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm> wrote:
>
> > the source logfile name is not included in the payload by the syslog
> spec.
> > It
> > may be in the case of your appliance, but we would need to see a sample
> > log to
> > understand ho to parse it.
> >
> > based on your template, you are using app-name, which may be listed
> > separtely if
> > it's a RFC5424 format log, or may be part of the syslog tag if it's a
> > RFC3164
> > format log over the wire (neither format has a way to specify a
> source log
> > file
> > by default)
> >
> > you can look at
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
> > and see the *-cc
> > options that you could apply to the app-name to eliminate control
> > characters.
> >
> > Again, we really need to see the original log message to understand
> what's
> > what.
> > Please log it with the templateRSYSLOG_DebugFormat so we can see exactly
> > what is
> > sent over the wire and how rsyslog has parsed it.
> >
> > David Lang
> >
> > On Tue, 23 Mar 2021, Scott Slattery via rsyslog
> > wrote:
> >
> > > Date: Tue, 23 Mar 2021 16:05:45 -0700
> > > From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
> > > To: John Chivian <jchivian@chivian.com>
> > > Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
> > > rsyslog-users <rsyslog@lists.adiscon.com>
> > > Subject: Re: [rsyslog] Altering forwarded logfile names
> > >
> > > Thanks, John, let me try to clarify what I mean.
> > >
> > > Normally when I forward from a remote server to the central log
> server, I
> > > can include a tag that can then be used to determine the file name
> I want
> > > on the central server. Since I have no real way to include this tag
> from
> > > the appliance, this is not an option.
> > >
> > > I'm looking for a way of inspecting the incoming packets to determining
> > the
> > > source logfile name (which is included in the payload) and use that
> > > filename on the target central server. Since there are multiple logs
> > being
> > > sent (access, audit, monitor, etc.), I'd like to segregate these into
> > their
> > > own files. I'm already using a template with the host information to
> > > dynamically create the file names. I just don't know how I can go
> beyond
> > > this to also include the source logname.
> > >
> > > Here's the template I'm using. It works for all other hosts where I can
> > > configure the tag but I get garbage names from the appliance. I had
> hoped
> > > that the appliance included some standard syslog tags but it
> doesn't seem
> > > so.
> > >
> > > template(name="DynRemoteLogFile" type="string"
> > >
> >
> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
> > >
> > > *Scott Slattery*
> > >
> > > *Sr. Systems & Cloud Architect*
> > >
> > > *Cloud, Compute, Information & Architecture Team*
> > >
> > > motorolasolutions.com
> > >
> > > *O: 602.529.8226*
> > >
> > > *E*: Scott.Slattery@MotorolaSolutions.com
> > >
> > >
> > >
> > >
> > > On Tue, Mar 23, 2021 at 3:30 PM John Chivian <jchivian@chivian.com>
> > wrote:
> > >
> > >> Your use of the term “file name” is confusing. When senders deliver to
> > >> rsyslog over the network there is no exchange of files or filenames,
> > only
> > >> packets of information. Those packets are expected to be in a format
> > that
> > >> syslog understands such that useful information (header elements and
> > >> message body) may be parsed from them. If you as the rsyslog admin
> > choose
> > >> to use some of that header information to compose filenames for output
> > >> files, then yes you are sort of at the mercy of the senders content
> > >> (especially if the sender doesn’t follow the syslog rules). However,
> > there
> > >> are functions in the advanced syntax that can be used to perform the
> > type
> > >> of character replacements you’re talking about.
> > >>
> > >> It is common practice to use the syslog header/rsyslog property
> element
> > >> called “hostname” for just such purposes. Is this what you’re talking
> > >> about? You’d have to provide your configuration for real analysis, at
> > >> least the part you perceive to be responsible for the problem.
> > >>
> > >> Regards,
> > >>
> > >>
> > >>
> > >> > On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
> > >> rsyslog@lists.adiscon.com> wrote:
> > >> >
> > >> > I have a configured central log collector using rsyslog. A few
> of the
> > >> > devices forwarding their logs are appliances that have no
> > configuration
> > >> > options other than the IP forwarding address and protocol. I cannot
> > >> control
> > >> > what file names are being sent.
> > >> >
> > >> > Unfortunately, they are sending unintelligible file names with
> > characters
> > >> > that normally would be escaped. Is there any way I can control or
> > alter
> > >> the
> > >> > incoming file name to normalize it to avoid these odd characters?
> > >> >
> > >> > For example, could I establish a character map that maps the
> unallowed
> > >> > character to something acceptable?
> > >> >
> > >> > thanks,
> > >> >
> > >> > *Scott Slattery*
> > >> >
> > >> > *Sr. Systems & Cloud Architect*
> > >> >
> > >> > *Cloud, Compute, Information & Architecture Team*
> > >> >
> > >> > motorolasolutions.com
> > >> >
> > >> > *O: 602.529.822*
> > >> >
> > >> > *E*: Scott.Slattery@MotorolaSolutions.com
> > >> >
> > >> > --
> > >> >
> > >> >
> > >> > *For more information on how and why we collect your personal
> > >> > information, please visit our Privacy Policy
> > >> > <
> > >>
> >
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> > >> >.*
> > >> > _______________________________________________
> > >> > rsyslog mailing list
> > >> >
> > >>
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
> > >> >
> > >>
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
> > >> > What's up with rsyslog? Follow
> > >>
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
> > >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad
> > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > >> DON'T LIKE THAT.
> > >>
> > >>
> > >
> > > --
> > >
> > >
> > > *For more information on how and why we collect your personal
> > > information, please visit our Privacy Policy
> > > <
> >
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> > >.*
> > > _______________________________________________
> > > rsyslog mailing list
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
> > > What's up with rsyslog? Follow
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
Yes, all the logs are coming in this way. There are three separate logs
being sent and each is incorrect in its own right but the same format
behavior is consistent across each of the three logs. I will look further
into the suggestions you have provided and have already reached out to the
appliance support team and will raise the nonconformance with the RFC.

This does provide me some valuable guidance and I thank each of you for
your feedback.

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: Scott.Slattery@MotorolaSolutions.com




On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com> wrote:

> There's something strange with this log entry. Notice that it looks as if
> it were more than one log entries joined with LF character (escaped on
> input as #012). And all "further" entries start with a - still
> nonconformant to the RFC but "fuller" - timestamp containing year.It's the
> leading event that seems to be trimmed in the front somehow. Do all events
> arrive like this?Wys?ano z telefonu Samsung<div>
> </div><div>
> </div><!-- originalMessage --><div>-------- Original message
> --------</div><div>From: Scott Slattery via rsyslog <
> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021 01:52
> (GMT+01:00) </div><div>To: David Lang <david@lang.hm> </div><div>Cc:
> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott Slattery via
> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re: [rsyslog]
> Altering forwarded logfile names </div><div>
> </div>
>
> Hi David, fortunately I had already done this. I'm including an actual log
> entry but have anonymized the data to keep the actual user and email
> address confidential:
>
> Debug line with all properties:
> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
> , fromhost-ip: '10.41.102.143'
> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
> , PRI: 13,
> syslogtag '03-23T16:'
> , programname: '03-23T16'
> , APP-NAME: '03-23T16'
> , PROCID: '-'
> , MSGID: '-'
> , TIMNESTAMP: 'Mar 23 21:47:13'
> , STRUCTURED-DATA: '-'
> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY
> ACCESS AUTHZ SESSION INFO USER_SESSION
> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> apigniodashboard-uat.mycompany.com"
> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
> UserName=Joe.User@mycompantions.com
>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="Joe.User@mycompan'
> *escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> apigniodashboard-uat.mycompany.com"
> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
> UserName=Joe.User@mycompantions.com
>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="Joe.User@mycompan'
> *rawmsg*: '03-23T16:47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> apigniodashboard-uat.mycompany.com"
> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
> UserName=Joe.User@mycompantions.com
>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="Joe.User@mycompan'
>
>
> By inspecting the rawmsg, I can see that field four (space-delimited)
> indicates this is the ACCESS log. So if I were able to extract the log
> identifier from the msg, I could then write all access logs to the same
> daily file. There are other formats as well from the same device but the
> idea is the same.
>
> *Scott Slattery*
>
> *Sr. Systems & Cloud Architect*
>
> *Cloud, Compute, Information & Architecture Team*
>
> motorolasolutions.com
>
> *O: 602.529.8226*
>
> *E*: Scott.Slattery@MotorolaSolutions.com
>
>
>
>
> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm> wrote:
>
> > the source logfile name is not included in the payload by the syslog
> spec.
> > It
> > may be in the case of your appliance, but we would need to see a sample
> > log to
> > understand ho to parse it.
> >
> > based on your template, you are using app-name, which may be listed
> > separtely if
> > it's a RFC5424 format log, or may be part of the syslog tag if it's a
> > RFC3164
> > format log over the wire (neither format has a way to specify a source
> log
> > file
> > by default)
> >
> > you can look at
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
> > and see the *-cc
> > options that you could apply to the app-name to eliminate control
> > characters.
> >
> > Again, we really need to see the original log message to understand
> what's
> > what.
> > Please log it with the templateRSYSLOG_DebugFormat so we can see exactly
> > what is
> > sent over the wire and how rsyslog has parsed it.
> >
> > David Lang
> >
> > On Tue, 23 Mar 2021, Scott Slattery via rsyslog
> > wrote:
> >
> > > Date: Tue, 23 Mar 2021 16:05:45 -0700
> > > From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
> > > To: John Chivian <jchivian@chivian.com>
> > > Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
> > > rsyslog-users <rsyslog@lists.adiscon.com>
> > > Subject: Re: [rsyslog] Altering forwarded logfile names
> > >
> > > Thanks, John, let me try to clarify what I mean.
> > >
> > > Normally when I forward from a remote server to the central log
> server, I
> > > can include a tag that can then be used to determine the file name I
> want
> > > on the central server. Since I have no real way to include this tag
> from
> > > the appliance, this is not an option.
> > >
> > > I'm looking for a way of inspecting the incoming packets to determining
> > the
> > > source logfile name (which is included in the payload) and use that
> > > filename on the target central server. Since there are multiple logs
> > being
> > > sent (access, audit, monitor, etc.), I'd like to segregate these into
> > their
> > > own files. I'm already using a template with the host information to
> > > dynamically create the file names. I just don't know how I can go
> beyond
> > > this to also include the source logname.
> > >
> > > Here's the template I'm using. It works for all other hosts where I can
> > > configure the tag but I get garbage names from the appliance. I had
> hoped
> > > that the appliance included some standard syslog tags but it doesn't
> seem
> > > so.
> > >
> > > template(name="DynRemoteLogFile" type="string"
> > >
> >
> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
> > >
> > > *Scott Slattery*
> > >
> > > *Sr. Systems & Cloud Architect*
> > >
> > > *Cloud, Compute, Information & Architecture Team*
> > >
> > > motorolasolutions.com
> > >
> > > *O: 602.529.8226*
> > >
> > > *E*: Scott.Slattery@MotorolaSolutions.com
> > >
> > >
> > >
> > >
> > > On Tue, Mar 23, 2021 at 3:30 PM John Chivian <jchivian@chivian.com>
> > wrote:
> > >
> > >> Your use of the term “file name” is confusing. When senders deliver
> to
> > >> rsyslog over the network there is no exchange of files or filenames,
> > only
> > >> packets of information. Those packets are expected to be in a format
> > that
> > >> syslog understands such that useful information (header elements and
> > >> message body) may be parsed from them. If you as the rsyslog admin
> > choose
> > >> to use some of that header information to compose filenames for output
> > >> files, then yes you are sort of at the mercy of the senders content
> > >> (especially if the sender doesn’t follow the syslog rules). However,
> > there
> > >> are functions in the advanced syntax that can be used to perform the
> > type
> > >> of character replacements you’re talking about.
> > >>
> > >> It is common practice to use the syslog header/rsyslog property
> element
> > >> called “hostname” for just such purposes. Is this what you’re talking
> > >> about? You’d have to provide your configuration for real analysis, at
> > >> least the part you perceive to be responsible for the problem.
> > >>
> > >> Regards,
> > >>
> > >>
> > >>
> > >> > On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
> > >> rsyslog@lists.adiscon.com> wrote:
> > >> >
> > >> > I have a configured central log collector using rsyslog. A few of
> the
> > >> > devices forwarding their logs are appliances that have no
> > configuration
> > >> > options other than the IP forwarding address and protocol. I cannot
> > >> control
> > >> > what file names are being sent.
> > >> >
> > >> > Unfortunately, they are sending unintelligible file names with
> > characters
> > >> > that normally would be escaped. Is there any way I can control or
> > alter
> > >> the
> > >> > incoming file name to normalize it to avoid these odd characters?
> > >> >
> > >> > For example, could I establish a character map that maps the
> unallowed
> > >> > character to something acceptable?
> > >> >
> > >> > thanks,
> > >> >
> > >> > *Scott Slattery*
> > >> >
> > >> > *Sr. Systems & Cloud Architect*
> > >> >
> > >> > *Cloud, Compute, Information & Architecture Team*
> > >> >
> > >> > motorolasolutions.com
> > >> >
> > >> > *O: 602.529.822*
> > >> >
> > >> > *E*: Scott.Slattery@MotorolaSolutions.com
> > >> >
> > >> > --
> > >> >
> > >> >
> > >> > *For more information on how and why we collect your personal
> > >> > information, please visit our Privacy Policy
> > >> > <
> > >>
> >
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> > >> >.*
> > >> > _______________________________________________
> > >> > rsyslog mailing list
> > >> >
> > >>
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
> > >> >
> > >>
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
> > >> > What's up with rsyslog? Follow
> > >>
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
> > >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad
> > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > >> DON'T LIKE THAT.
> > >>
> > >>
> > >
> > > --
> > >
> > >
> > > *For more information on how and why we collect your personal
> > > information, please visit our Privacy Policy
> > > <
> >
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> > >.*
> > > _______________________________________________
> > > rsyslog mailing list
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
> > > What's up with rsyslog? Follow
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
>
> --
>
>
> *For more information on how and why we collect your personal
> information, please visit our Privacy Policy
> <
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >.*
> _______________________________________________
> rsyslog mailing list
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
> What's up with rsyslog? Follow
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.

--


*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
when you say 'each is incorrect but the same format'

does that mean that every log has the year missing? or that every log is
combining the logs together?

I'll note that it's possible to define a custom parser using the mmnormalize
library and add it to the parsing stack. I helped define the need for such a
capability, but haven't used it yet.

David Lang

On Wed, 24 Mar 2021, Scott Slattery wrote:

> Date: Wed, 24 Mar 2021 08:58:07 -0700
> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> To: mariusz.kruk@safecomp.com
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <david@lang.hm>
> Subject: Re: [rsyslog] Altering forwarded logfile names
>
> Yes, all the logs are coming in this way. There are three separate logs
> being sent and each is incorrect in its own right but the same format
> behavior is consistent across each of the three logs. I will look further
> into the suggestions you have provided and have already reached out to the
> appliance support team and will raise the nonconformance with the RFC.
>
> This does provide me some valuable guidance and I thank each of you for
> your feedback.
>
> *Scott Slattery*
>
> *Sr. Systems & Cloud Architect*
>
> *Cloud, Compute, Information & Architecture Team*
>
> motorolasolutions.com
>
> *O: 602.529.8226*
>
> *E*: Scott.Slattery@MotorolaSolutions.com
>
>
>
>
> On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com> wrote:
>
>> There's something strange with this log entry. Notice that it looks as if
>> it were more than one log entries joined with LF character (escaped on
>> input as #012). And all "further" entries start with a - still
>> nonconformant to the RFC but "fuller" - timestamp containing year.It's the
>> leading event that seems to be trimmed in the front somehow. Do all events
>> arrive like this?Wys?ano z telefonu Samsung<div>
>> </div><div>
>> </div><!-- originalMessage --><div>-------- Original message
>> --------</div><div>From: Scott Slattery via rsyslog <
>> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021 01:52
>> (GMT+01:00) </div><div>To: David Lang <david@lang.hm> </div><div>Cc:
>> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott Slattery via
>> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re: [rsyslog]
>> Altering forwarded logfile names </div><div>
>> </div>
>>
>> Hi David, fortunately I had already done this. I'm including an actual log
>> entry but have anonymized the data to keep the actual user and email
>> address confidential:
>>
>> Debug line with all properties:
>> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
>> , fromhost-ip: '10.41.102.143'
>> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
>> , PRI: 13,
>> syslogtag '03-23T16:'
>> , programname: '03-23T16'
>> , APP-NAME: '03-23T16'
>> , PROCID: '-'
>> , MSGID: '-'
>> , TIMNESTAMP: 'Mar 23 21:47:13'
>> , STRUCTURED-DATA: '-'
>> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY
>> ACCESS AUTHZ SESSION INFO USER_SESSION
>> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>> apigniodashboard-uat.mycompany.com"
>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
>> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
>> Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
>> UserName=Joe.User@mycompantions.com
>>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>> Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
>> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> SUBJECT="Joe.User@mycompan'
>> *escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
>> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
>> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>> apigniodashboard-uat.mycompany.com"
>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
>> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
>> Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
>> UserName=Joe.User@mycompantions.com
>>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>> Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
>> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> SUBJECT="Joe.User@mycompan'
>> *rawmsg*: '03-23T16:47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
>> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
>> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>> apigniodashboard-uat.mycompany.com"
>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
>> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
>> Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
>> UserName=Joe.User@mycompantions.com
>>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>> Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
>> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> SUBJECT="Joe.User@mycompan'
>>
>>
>> By inspecting the rawmsg, I can see that field four (space-delimited)
>> indicates this is the ACCESS log. So if I were able to extract the log
>> identifier from the msg, I could then write all access logs to the same
>> daily file. There are other formats as well from the same device but the
>> idea is the same.
>>
>> *Scott Slattery*
>>
>> *Sr. Systems & Cloud Architect*
>>
>> *Cloud, Compute, Information & Architecture Team*
>>
>> motorolasolutions.com
>>
>> *O: 602.529.8226*
>>
>> *E*: Scott.Slattery@MotorolaSolutions.com
>>
>>
>>
>>
>> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm> wrote:
>>
>>> the source logfile name is not included in the payload by the syslog
>> spec.
>>> It
>>> may be in the case of your appliance, but we would need to see a sample
>>> log to
>>> understand ho to parse it.
>>>
>>> based on your template, you are using app-name, which may be listed
>>> separtely if
>>> it's a RFC5424 format log, or may be part of the syslog tag if it's a
>>> RFC3164
>>> format log over the wire (neither format has a way to specify a source
>> log
>>> file
>>> by default)
>>>
>>> you can look at
>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
>>> and see the *-cc
>>> options that you could apply to the app-name to eliminate control
>>> characters.
>>>
>>> Again, we really need to see the original log message to understand
>> what's
>>> what.
>>> Please log it with the templateRSYSLOG_DebugFormat so we can see exactly
>>> what is
>>> sent over the wire and how rsyslog has parsed it.
>>>
>>> David Lang
>>>
>>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
>>> wrote:
>>>
>>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
>>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
>>>> To: John Chivian <jchivian@chivian.com>
>>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
>>>> rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>>
>>>> Thanks, John, let me try to clarify what I mean.
>>>>
>>>> Normally when I forward from a remote server to the central log
>> server, I
>>>> can include a tag that can then be used to determine the file name I
>> want
>>>> on the central server. Since I have no real way to include this tag
>> from
>>>> the appliance, this is not an option.
>>>>
>>>> I'm looking for a way of inspecting the incoming packets to determining
>>> the
>>>> source logfile name (which is included in the payload) and use that
>>>> filename on the target central server. Since there are multiple logs
>>> being
>>>> sent (access, audit, monitor, etc.), I'd like to segregate these into
>>> their
>>>> own files. I'm already using a template with the host information to
>>>> dynamically create the file names. I just don't know how I can go
>> beyond
>>>> this to also include the source logname.
>>>>
>>>> Here's the template I'm using. It works for all other hosts where I can
>>>> configure the tag but I get garbage names from the appliance. I had
>> hoped
>>>> that the appliance included some standard syslog tags but it doesn't
>> seem
>>>> so.
>>>>
>>>> template(name="DynRemoteLogFile" type="string"
>>>>
>>>
>> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
>>>>
>>>> *Scott Slattery*
>>>>
>>>> *Sr. Systems & Cloud Architect*
>>>>
>>>> *Cloud, Compute, Information & Architecture Team*
>>>>
>>>> motorolasolutions.com
>>>>
>>>> *O: 602.529.8226*
>>>>
>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <jchivian@chivian.com>
>>> wrote:
>>>>
>>>>> Your use of the term “file name” is confusing. When senders deliver
>> to
>>>>> rsyslog over the network there is no exchange of files or filenames,
>>> only
>>>>> packets of information. Those packets are expected to be in a format
>>> that
>>>>> syslog understands such that useful information (header elements and
>>>>> message body) may be parsed from them. If you as the rsyslog admin
>>> choose
>>>>> to use some of that header information to compose filenames for output
>>>>> files, then yes you are sort of at the mercy of the senders content
>>>>> (especially if the sender doesn’t follow the syslog rules). However,
>>> there
>>>>> are functions in the advanced syntax that can be used to perform the
>>> type
>>>>> of character replacements you’re talking about.
>>>>>
>>>>> It is common practice to use the syslog header/rsyslog property
>> element
>>>>> called “hostname” for just such purposes. Is this what you’re talking
>>>>> about? You’d have to provide your configuration for real analysis, at
>>>>> least the part you perceive to be responsible for the problem.
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>>
>>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
>>>>> rsyslog@lists.adiscon.com> wrote:
>>>>>>
>>>>>> I have a configured central log collector using rsyslog. A few of
>> the
>>>>>> devices forwarding their logs are appliances that have no
>>> configuration
>>>>>> options other than the IP forwarding address and protocol. I cannot
>>>>> control
>>>>>> what file names are being sent.
>>>>>>
>>>>>> Unfortunately, they are sending unintelligible file names with
>>> characters
>>>>>> that normally would be escaped. Is there any way I can control or
>>> alter
>>>>> the
>>>>>> incoming file name to normalize it to avoid these odd characters?
>>>>>>
>>>>>> For example, could I establish a character map that maps the
>> unallowed
>>>>>> character to something acceptable?
>>>>>>
>>>>>> thanks,
>>>>>>
>>>>>> *Scott Slattery*
>>>>>>
>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>
>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>
>>>>>> motorolasolutions.com
>>>>>>
>>>>>> *O: 602.529.822*
>>>>>>
>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>
>>>>>> --
>>>>>>
>>>>>>
>>>>>> *For more information on how and why we collect your personal
>>>>>> information, please visit our Privacy Policy
>>>>>> <
>>>>>
>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>> .*
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>>
>>>>>
>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
>>>>>>
>>>>>
>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
>>>>>> What's up with rsyslog? Follow
>>>>>
>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>>
>>>> *For more information on how and why we collect your personal
>>>> information, please visit our Privacy Policy
>>>> <
>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>> .*
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>>
>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
>>>>
>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
>>>> What's up with rsyslog? Follow
>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>
>> --
>>
>>
>> *For more information on how and why we collect your personal
>> information, please visit our Privacy Policy
>> <
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>> .*
>> _______________________________________________
>> rsyslog mailing list
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
>> What's up with rsyslog? Follow
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
From what I can discern since there are lots of transactions, there seem to
be multiple payload formats. For all that I have so far analyzed, the date
truncation issue is the same regardless of the payload received. Like you,
I've never used mmnormalize but I will look at it as an option. I've
reached out to the product support team to better understand if there is a
appliance-side misconfiguration and something that can be corrected there
or if I need to make adjustments on the rsyslog side to clean up the
records.

What I meant is that there are multiple message formats (payload data) but
there seems to be a consistent truncation of the date field just as you saw
in the app-name tag field.

I appreciate your ongoing feedback and help. This issue is new to me and
will need a creative solution if the vendor has no answers.

Many thanks,


*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: Scott.Slattery@MotorolaSolutions.com




On Wed, Mar 24, 2021 at 2:14 PM David Lang <david@lang.hm> wrote:

> when you say 'each is incorrect but the same format'
>
> does that mean that every log has the year missing? or that every log is
> combining the logs together?
>
> I'll note that it's possible to define a custom parser using the
> mmnormalize
> library and add it to the parsing stack. I helped define the need for such
> a
> capability, but haven't used it yet.
>
> David Lang
>
> On Wed, 24 Mar 2021, Scott Slattery wrote:
>
> > Date: Wed, 24 Mar 2021 08:58:07 -0700
> > From: Scott Slattery <scott.slattery@motorolasolutions.com>
> > To: mariusz.kruk@safecomp.com
> > Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <david@lang.hm
> >
> > Subject: Re: [rsyslog] Altering forwarded logfile names
> >
> > Yes, all the logs are coming in this way. There are three separate logs
> > being sent and each is incorrect in its own right but the same format
> > behavior is consistent across each of the three logs. I will look further
> > into the suggestions you have provided and have already reached out to
> the
> > appliance support team and will raise the nonconformance with the RFC.
> >
> > This does provide me some valuable guidance and I thank each of you for
> > your feedback.
> >
> > *Scott Slattery*
> >
> > *Sr. Systems & Cloud Architect*
> >
> > *Cloud, Compute, Information & Architecture Team*
> >
> > motorolasolutions.com
> >
> > *O: 602.529.8226*
> >
> > *E*: Scott.Slattery@MotorolaSolutions.com
> >
> >
> >
> >
> > On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com> wrote:
> >
> >> There's something strange with this log entry. Notice that it looks as
> if
> >> it were more than one log entries joined with LF character (escaped on
> >> input as #012). And all "further" entries start with a - still
> >> nonconformant to the RFC but "fuller" - timestamp containing year.It's
> the
> >> leading event that seems to be trimmed in the front somehow. Do all
> events
> >> arrive like this?Wys?ano z telefonu Samsung<div>
> >> </div><div>
> >> </div><!-- originalMessage --><div>-------- Original message
> >> --------</div><div>From: Scott Slattery via rsyslog <
> >> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021 01:52
> >> (GMT+01:00) </div><div>To: David Lang <david@lang.hm> </div><div>Cc:
> >> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott Slattery
> via
> >> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re: [rsyslog]
> >> Altering forwarded logfile names </div><div>
> >> </div>
> >>
> >> Hi David, fortunately I had already done this. I'm including an actual
> log
> >> entry but have anonymized the data to keep the actual user and email
> >> address confidential:
> >>
> >> Debug line with all properties:
> >> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
> >> , fromhost-ip: '10.41.102.143'
> >> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
> >> , PRI: 13,
> >> syslogtag '03-23T16:'
> >> , programname: '03-23T16'
> >> , APP-NAME: '03-23T16'
> >> , PROCID: '-'
> >> , MSGID: '-'
> >> , TIMNESTAMP: 'Mar 23 21:47:13'
> >> , STRUCTURED-DATA: '-'
> >> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> ACCESS_GATEWAY
> >> ACCESS AUTHZ SESSION INFO USER_SESSION
> >> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> >> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >> apigniodashboard-uat.mycompany.com"
> >> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> REMOTE_IP="10.44.65.38"
> >> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36"]
> >> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
> INFO
> >> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> >> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> APP="Ignio
> >> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> >> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> >> Joe.User@mycompantions.com RelayDomain=
> apigniodashboard-uat.mycompany.com
> >> UserName=Joe.User@mycompantions.com
> >>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
> >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> >> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac
> OS X
> >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >> Safari/537.36"] allow access to
> resource#0122021-03-23T16:47:20.711-05:00
> >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION
> INFO
> >> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >> SUBJECT="Joe.User@mycompan'
> >> *escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> >> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
> >> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> >> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >> apigniodashboard-uat.mycompany.com"
> >> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> REMOTE_IP="10.44.65.38"
> >> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36"]
> >> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
> INFO
> >> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> >> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> APP="Ignio
> >> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> >> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> >> Joe.User@mycompantions.com RelayDomain=
> apigniodashboard-uat.mycompany.com
> >> UserName=Joe.User@mycompantions.com
> >>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
> >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> >> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac
> OS X
> >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >> Safari/537.36"] allow access to
> resource#0122021-03-23T16:47:20.711-05:00
> >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION
> INFO
> >> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >> SUBJECT="Joe.User@mycompan'
> >> *rawmsg*: '03-23T16:47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> >> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
> >> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> >> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >> apigniodashboard-uat.mycompany.com"
> >> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> REMOTE_IP="10.44.65.38"
> >> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> Safari/537.36"]
> >> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
> INFO
> >> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> >> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> APP="Ignio
> >> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> >> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> >> Joe.User@mycompantions.com RelayDomain=
> apigniodashboard-uat.mycompany.com
> >> UserName=Joe.User@mycompantions.com
> >>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
> >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> >> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac
> OS X
> >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >> Safari/537.36"] allow access to
> resource#0122021-03-23T16:47:20.711-05:00
> >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION
> INFO
> >> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >> SUBJECT="Joe.User@mycompan'
> >>
> >>
> >> By inspecting the rawmsg, I can see that field four (space-delimited)
> >> indicates this is the ACCESS log. So if I were able to extract the log
> >> identifier from the msg, I could then write all access logs to the same
> >> daily file. There are other formats as well from the same device but the
> >> idea is the same.
> >>
> >> *Scott Slattery*
> >>
> >> *Sr. Systems & Cloud Architect*
> >>
> >> *Cloud, Compute, Information & Architecture Team*
> >>
> >> motorolasolutions.com
> >>
> >> *O: 602.529.8226*
> >>
> >> *E*: Scott.Slattery@MotorolaSolutions.com
> >>
> >>
> >>
> >>
> >> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm> wrote:
> >>
> >>> the source logfile name is not included in the payload by the syslog
> >> spec.
> >>> It
> >>> may be in the case of your appliance, but we would need to see a sample
> >>> log to
> >>> understand ho to parse it.
> >>>
> >>> based on your template, you are using app-name, which may be listed
> >>> separtely if
> >>> it's a RFC5424 format log, or may be part of the syslog tag if it's a
> >>> RFC3164
> >>> format log over the wire (neither format has a way to specify a source
> >> log
> >>> file
> >>> by default)
> >>>
> >>> you can look at
> >>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
> >>> and see the *-cc
> >>> options that you could apply to the app-name to eliminate control
> >>> characters.
> >>>
> >>> Again, we really need to see the original log message to understand
> >> what's
> >>> what.
> >>> Please log it with the templateRSYSLOG_DebugFormat so we can see
> exactly
> >>> what is
> >>> sent over the wire and how rsyslog has parsed it.
> >>>
> >>> David Lang
> >>>
> >>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
> >>> wrote:
> >>>
> >>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
> >>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
> >>>> To: John Chivian <jchivian@chivian.com>
> >>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
> >>>> rsyslog-users <rsyslog@lists.adiscon.com>
> >>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>>
> >>>> Thanks, John, let me try to clarify what I mean.
> >>>>
> >>>> Normally when I forward from a remote server to the central log
> >> server, I
> >>>> can include a tag that can then be used to determine the file name I
> >> want
> >>>> on the central server. Since I have no real way to include this tag
> >> from
> >>>> the appliance, this is not an option.
> >>>>
> >>>> I'm looking for a way of inspecting the incoming packets to
> determining
> >>> the
> >>>> source logfile name (which is included in the payload) and use that
> >>>> filename on the target central server. Since there are multiple logs
> >>> being
> >>>> sent (access, audit, monitor, etc.), I'd like to segregate these into
> >>> their
> >>>> own files. I'm already using a template with the host information to
> >>>> dynamically create the file names. I just don't know how I can go
> >> beyond
> >>>> this to also include the source logname.
> >>>>
> >>>> Here's the template I'm using. It works for all other hosts where I
> can
> >>>> configure the tag but I get garbage names from the appliance. I had
> >> hoped
> >>>> that the appliance included some standard syslog tags but it doesn't
> >> seem
> >>>> so.
> >>>>
> >>>> template(name="DynRemoteLogFile" type="string"
> >>>>
> >>>
> >>
> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
> >>>>
> >>>> *Scott Slattery*
> >>>>
> >>>> *Sr. Systems & Cloud Architect*
> >>>>
> >>>> *Cloud, Compute, Information & Architecture Team*
> >>>>
> >>>> motorolasolutions.com
> >>>>
> >>>> *O: 602.529.8226*
> >>>>
> >>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <jchivian@chivian.com>
> >>> wrote:
> >>>>
> >>>>> Your use of the term “file name” is confusing. When senders deliver
> >> to
> >>>>> rsyslog over the network there is no exchange of files or filenames,
> >>> only
> >>>>> packets of information. Those packets are expected to be in a format
> >>> that
> >>>>> syslog understands such that useful information (header elements and
> >>>>> message body) may be parsed from them. If you as the rsyslog admin
> >>> choose
> >>>>> to use some of that header information to compose filenames for
> output
> >>>>> files, then yes you are sort of at the mercy of the senders content
> >>>>> (especially if the sender doesn’t follow the syslog rules). However,
> >>> there
> >>>>> are functions in the advanced syntax that can be used to perform the
> >>> type
> >>>>> of character replacements you’re talking about.
> >>>>>
> >>>>> It is common practice to use the syslog header/rsyslog property
> >> element
> >>>>> called “hostname” for just such purposes. Is this what you’re
> talking
> >>>>> about? You’d have to provide your configuration for real analysis,
> at
> >>>>> least the part you perceive to be responsible for the problem.
> >>>>>
> >>>>> Regards,
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
> >>>>> rsyslog@lists.adiscon.com> wrote:
> >>>>>>
> >>>>>> I have a configured central log collector using rsyslog. A few of
> >> the
> >>>>>> devices forwarding their logs are appliances that have no
> >>> configuration
> >>>>>> options other than the IP forwarding address and protocol. I cannot
> >>>>> control
> >>>>>> what file names are being sent.
> >>>>>>
> >>>>>> Unfortunately, they are sending unintelligible file names with
> >>> characters
> >>>>>> that normally would be escaped. Is there any way I can control or
> >>> alter
> >>>>> the
> >>>>>> incoming file name to normalize it to avoid these odd characters?
> >>>>>>
> >>>>>> For example, could I establish a character map that maps the
> >> unallowed
> >>>>>> character to something acceptable?
> >>>>>>
> >>>>>> thanks,
> >>>>>>
> >>>>>> *Scott Slattery*
> >>>>>>
> >>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>
> >>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>
> >>>>>> motorolasolutions.com
> >>>>>>
> >>>>>> *O: 602.529.822*
> >>>>>>
> >>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>>
> >>>>>> *For more information on how and why we collect your personal
> >>>>>> information, please visit our Privacy Policy
> >>>>>> <
> >>>>>
> >>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>> .*
> >>>>>> _______________________________________________
> >>>>>> rsyslog mailing list
> >>>>>>
> >>>>>
> >>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
> >>>>>>
> >>>>>
> >>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
> >>>>>> What's up with rsyslog? Follow
> >>>>>
> >>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
> >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>> myriad
> >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> you
> >>>>> DON'T LIKE THAT.
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>>
> >>>>
> >>>> *For more information on how and why we collect your personal
> >>>> information, please visit our Privacy Policy
> >>>> <
> >>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>> .*
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>>
> >>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
> >>>>
> >>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
> >>>> What's up with rsyslog? Follow
> >>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >> myriad
> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >>> DON'T LIKE THAT.
> >>
> >> --
> >>
> >>
> >> *For more information on how and why we collect your personal
> >> information, please visit our Privacy Policy
> >> <
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>> .*
> >> _______________________________________________
> >> rsyslog mailing list
> >>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
> >>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
> >> What's up with rsyslog? Follow
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> DON'T LIKE THAT.
> >
> >

--


*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
to clarify, I have a lot of mmnormalize experience, what I don't have is
experience using the option to use mmnormalize to receive the message

the other issue where the message separator seems to be linefeed insted of
newline and is causing messages to run together.

David Lang

On Wed, 24 Mar 2021, Scott Slattery wrote:

> Date: Wed, 24 Mar 2021 14:23:07 -0700
> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> To: David Lang <david@lang.hm>
> Cc: mariusz.kruk@safecomp.com, rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] Altering forwarded logfile names
>
> From what I can discern since there are lots of transactions, there seem to
> be multiple payload formats. For all that I have so far analyzed, the date
> truncation issue is the same regardless of the payload received. Like you,
> I've never used mmnormalize but I will look at it as an option. I've
> reached out to the product support team to better understand if there is a
> appliance-side misconfiguration and something that can be corrected there
> or if I need to make adjustments on the rsyslog side to clean up the
> records.
>
> What I meant is that there are multiple message formats (payload data) but
> there seems to be a consistent truncation of the date field just as you saw
> in the app-name tag field.
>
> I appreciate your ongoing feedback and help. This issue is new to me and
> will need a creative solution if the vendor has no answers.
>
> Many thanks,
>
>
> *Scott Slattery*
>
> *Sr. Systems & Cloud Architect*
>
> *Cloud, Compute, Information & Architecture Team*
>
> motorolasolutions.com
>
> *O: 602.529.8226*
>
> *E*: Scott.Slattery@MotorolaSolutions.com
>
>
>
>
> On Wed, Mar 24, 2021 at 2:14 PM David Lang <david@lang.hm> wrote:
>
>> when you say 'each is incorrect but the same format'
>>
>> does that mean that every log has the year missing? or that every log is
>> combining the logs together?
>>
>> I'll note that it's possible to define a custom parser using the
>> mmnormalize
>> library and add it to the parsing stack. I helped define the need for such
>> a
>> capability, but haven't used it yet.
>>
>> David Lang
>>
>> On Wed, 24 Mar 2021, Scott Slattery wrote:
>>
>>> Date: Wed, 24 Mar 2021 08:58:07 -0700
>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>> To: mariusz.kruk@safecomp.com
>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <david@lang.hm
>>>
>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>
>>> Yes, all the logs are coming in this way. There are three separate logs
>>> being sent and each is incorrect in its own right but the same format
>>> behavior is consistent across each of the three logs. I will look further
>>> into the suggestions you have provided and have already reached out to
>> the
>>> appliance support team and will raise the nonconformance with the RFC.
>>>
>>> This does provide me some valuable guidance and I thank each of you for
>>> your feedback.
>>>
>>> *Scott Slattery*
>>>
>>> *Sr. Systems & Cloud Architect*
>>>
>>> *Cloud, Compute, Information & Architecture Team*
>>>
>>> motorolasolutions.com
>>>
>>> *O: 602.529.8226*
>>>
>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>
>>>
>>>
>>>
>>> On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com> wrote:
>>>
>>>> There's something strange with this log entry. Notice that it looks as
>> if
>>>> it were more than one log entries joined with LF character (escaped on
>>>> input as #012). And all "further" entries start with a - still
>>>> nonconformant to the RFC but "fuller" - timestamp containing year.It's
>> the
>>>> leading event that seems to be trimmed in the front somehow. Do all
>> events
>>>> arrive like this?Wys?ano z telefonu Samsung<div>
>>>> </div><div>
>>>> </div><!-- originalMessage --><div>-------- Original message
>>>> --------</div><div>From: Scott Slattery via rsyslog <
>>>> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021 01:52
>>>> (GMT+01:00) </div><div>To: David Lang <david@lang.hm> </div><div>Cc:
>>>> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott Slattery
>> via
>>>> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re: [rsyslog]
>>>> Altering forwarded logfile names </div><div>
>>>> </div>
>>>>
>>>> Hi David, fortunately I had already done this. I'm including an actual
>> log
>>>> entry but have anonymized the data to keep the actual user and email
>>>> address confidential:
>>>>
>>>> Debug line with all properties:
>>>> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
>>>> , fromhost-ip: '10.41.102.143'
>>>> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
>>>> , PRI: 13,
>>>> syslogtag '03-23T16:'
>>>> , programname: '03-23T16'
>>>> , APP-NAME: '03-23T16'
>>>> , PROCID: '-'
>>>> , MSGID: '-'
>>>> , TIMNESTAMP: 'Mar 23 21:47:13'
>>>> , STRUCTURED-DATA: '-'
>>>> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
>> ACCESS_GATEWAY
>>>> ACCESS AUTHZ SESSION INFO USER_SESSION
>>>> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>> apigniodashboard-uat.mycompany.com"
>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>> REMOTE_IP="10.44.65.38"
>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>> Safari/537.36"]
>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
>> INFO
>>>> USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
>> APP="Ignio
>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
>>>> Joe.User@mycompantions.com RelayDomain=
>> apigniodashboard-uat.mycompany.com
>>>> UserName=Joe.User@mycompantions.com
>>>>
>>>>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac
>> OS X
>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>> Safari/537.36"] allow access to
>> resource#0122021-03-23T16:47:20.711-05:00
>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION
>> INFO
>>>> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>> SUBJECT="Joe.User@mycompan'
>>>> *escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
>>>> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
>>>> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>> apigniodashboard-uat.mycompany.com"
>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>> REMOTE_IP="10.44.65.38"
>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>> Safari/537.36"]
>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
>> INFO
>>>> USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
>> APP="Ignio
>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
>>>> Joe.User@mycompantions.com RelayDomain=
>> apigniodashboard-uat.mycompany.com
>>>> UserName=Joe.User@mycompantions.com
>>>>
>>>>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac
>> OS X
>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>> Safari/537.36"] allow access to
>> resource#0122021-03-23T16:47:20.711-05:00
>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION
>> INFO
>>>> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>> SUBJECT="Joe.User@mycompan'
>>>> *rawmsg*: '03-23T16:47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
>>>> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
>>>> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>> apigniodashboard-uat.mycompany.com"
>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>> REMOTE_IP="10.44.65.38"
>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>> Safari/537.36"]
>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
>> INFO
>>>> USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
>> APP="Ignio
>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
>>>> Joe.User@mycompantions.com RelayDomain=
>> apigniodashboard-uat.mycompany.com
>>>> UserName=Joe.User@mycompantions.com
>>>>
>>>>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac
>> OS X
>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>> Safari/537.36"] allow access to
>> resource#0122021-03-23T16:47:20.711-05:00
>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION
>> INFO
>>>> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>> SUBJECT="Joe.User@mycompan'
>>>>
>>>>
>>>> By inspecting the rawmsg, I can see that field four (space-delimited)
>>>> indicates this is the ACCESS log. So if I were able to extract the log
>>>> identifier from the msg, I could then write all access logs to the same
>>>> daily file. There are other formats as well from the same device but the
>>>> idea is the same.
>>>>
>>>> *Scott Slattery*
>>>>
>>>> *Sr. Systems & Cloud Architect*
>>>>
>>>> *Cloud, Compute, Information & Architecture Team*
>>>>
>>>> motorolasolutions.com
>>>>
>>>> *O: 602.529.8226*
>>>>
>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm> wrote:
>>>>
>>>>> the source logfile name is not included in the payload by the syslog
>>>> spec.
>>>>> It
>>>>> may be in the case of your appliance, but we would need to see a sample
>>>>> log to
>>>>> understand ho to parse it.
>>>>>
>>>>> based on your template, you are using app-name, which may be listed
>>>>> separtely if
>>>>> it's a RFC5424 format log, or may be part of the syslog tag if it's a
>>>>> RFC3164
>>>>> format log over the wire (neither format has a way to specify a source
>>>> log
>>>>> file
>>>>> by default)
>>>>>
>>>>> you can look at
>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
>>>>> and see the *-cc
>>>>> options that you could apply to the app-name to eliminate control
>>>>> characters.
>>>>>
>>>>> Again, we really need to see the original log message to understand
>>>> what's
>>>>> what.
>>>>> Please log it with the templateRSYSLOG_DebugFormat so we can see
>> exactly
>>>>> what is
>>>>> sent over the wire and how rsyslog has parsed it.
>>>>>
>>>>> David Lang
>>>>>
>>>>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
>>>>> wrote:
>>>>>
>>>>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
>>>>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
>>>>>> To: John Chivian <jchivian@chivian.com>
>>>>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
>>>>>> rsyslog-users <rsyslog@lists.adiscon.com>
>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>>>>
>>>>>> Thanks, John, let me try to clarify what I mean.
>>>>>>
>>>>>> Normally when I forward from a remote server to the central log
>>>> server, I
>>>>>> can include a tag that can then be used to determine the file name I
>>>> want
>>>>>> on the central server. Since I have no real way to include this tag
>>>> from
>>>>>> the appliance, this is not an option.
>>>>>>
>>>>>> I'm looking for a way of inspecting the incoming packets to
>> determining
>>>>> the
>>>>>> source logfile name (which is included in the payload) and use that
>>>>>> filename on the target central server. Since there are multiple logs
>>>>> being
>>>>>> sent (access, audit, monitor, etc.), I'd like to segregate these into
>>>>> their
>>>>>> own files. I'm already using a template with the host information to
>>>>>> dynamically create the file names. I just don't know how I can go
>>>> beyond
>>>>>> this to also include the source logname.
>>>>>>
>>>>>> Here's the template I'm using. It works for all other hosts where I
>> can
>>>>>> configure the tag but I get garbage names from the appliance. I had
>>>> hoped
>>>>>> that the appliance included some standard syslog tags but it doesn't
>>>> seem
>>>>>> so.
>>>>>>
>>>>>> template(name="DynRemoteLogFile" type="string"
>>>>>>
>>>>>
>>>>
>> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
>>>>>>
>>>>>> *Scott Slattery*
>>>>>>
>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>
>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>
>>>>>> motorolasolutions.com
>>>>>>
>>>>>> *O: 602.529.8226*
>>>>>>
>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <jchivian@chivian.com>
>>>>> wrote:
>>>>>>
>>>>>>> Your use of the term “file name” is confusing. When senders deliver
>>>> to
>>>>>>> rsyslog over the network there is no exchange of files or filenames,
>>>>> only
>>>>>>> packets of information. Those packets are expected to be in a format
>>>>> that
>>>>>>> syslog understands such that useful information (header elements and
>>>>>>> message body) may be parsed from them. If you as the rsyslog admin
>>>>> choose
>>>>>>> to use some of that header information to compose filenames for
>> output
>>>>>>> files, then yes you are sort of at the mercy of the senders content
>>>>>>> (especially if the sender doesn’t follow the syslog rules). However,
>>>>> there
>>>>>>> are functions in the advanced syntax that can be used to perform the
>>>>> type
>>>>>>> of character replacements you’re talking about.
>>>>>>>
>>>>>>> It is common practice to use the syslog header/rsyslog property
>>>> element
>>>>>>> called “hostname” for just such purposes. Is this what you’re
>> talking
>>>>>>> about? You’d have to provide your configuration for real analysis,
>> at
>>>>>>> least the part you perceive to be responsible for the problem.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
>>>>>>> rsyslog@lists.adiscon.com> wrote:
>>>>>>>>
>>>>>>>> I have a configured central log collector using rsyslog. A few of
>>>> the
>>>>>>>> devices forwarding their logs are appliances that have no
>>>>> configuration
>>>>>>>> options other than the IP forwarding address and protocol. I cannot
>>>>>>> control
>>>>>>>> what file names are being sent.
>>>>>>>>
>>>>>>>> Unfortunately, they are sending unintelligible file names with
>>>>> characters
>>>>>>>> that normally would be escaped. Is there any way I can control or
>>>>> alter
>>>>>>> the
>>>>>>>> incoming file name to normalize it to avoid these odd characters?
>>>>>>>>
>>>>>>>> For example, could I establish a character map that maps the
>>>> unallowed
>>>>>>>> character to something acceptable?
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>>
>>>>>>>> *Scott Slattery*
>>>>>>>>
>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>
>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>
>>>>>>>> motorolasolutions.com
>>>>>>>>
>>>>>>>> *O: 602.529.822*
>>>>>>>>
>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>>
>>>>>>>> *For more information on how and why we collect your personal
>>>>>>>> information, please visit our Privacy Policy
>>>>>>>> <
>>>>>>>
>>>>>
>>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>>>> .*
>>>>>>>> _______________________________________________
>>>>>>>> rsyslog mailing list
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
>>>>>>>> What's up with rsyslog? Follow
>>>>>>>
>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>> myriad
>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you
>>>>>>> DON'T LIKE THAT.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>>
>>>>>> *For more information on how and why we collect your personal
>>>>>> information, please visit our Privacy Policy
>>>>>> <
>>>>>
>>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>> .*
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>>
>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
>>>>>>
>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
>>>>>> What's up with rsyslog? Follow
>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>> myriad
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>>
>>>> --
>>>>
>>>>
>>>> *For more information on how and why we collect your personal
>>>> information, please visit our Privacy Policy
>>>> <
>>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>> .*
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
>>>> What's up with rsyslog? Follow
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>
>>>
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
Sorry if I didn't catch it, but did you post the configuration? If
not, it would probably be useful.

Looking at the message properties, this looks like the message was
received via UDP but had a LF inside it which was intended as a frame
separator. If so, that is another bug in the vendor's implementation.
Framing is used for TCP, but not UDP. For UDP it is strictly one
message per package.

HTH,
Rainer

El mié, 24 mar 2021 a las 22:23, Scott Slattery via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> From what I can discern since there are lots of transactions, there seem to
> be multiple payload formats. For all that I have so far analyzed, the date
> truncation issue is the same regardless of the payload received. Like you,
> I've never used mmnormalize but I will look at it as an option. I've
> reached out to the product support team to better understand if there is a
> appliance-side misconfiguration and something that can be corrected there
> or if I need to make adjustments on the rsyslog side to clean up the
> records.
>
> What I meant is that there are multiple message formats (payload data) but
> there seems to be a consistent truncation of the date field just as you saw
> in the app-name tag field.
>
> I appreciate your ongoing feedback and help. This issue is new to me and
> will need a creative solution if the vendor has no answers.
>
> Many thanks,
>
>
> *Scott Slattery*
>
> *Sr. Systems & Cloud Architect*
>
> *Cloud, Compute, Information & Architecture Team*
>
> motorolasolutions.com
>
> *O: 602.529.8226*
>
> *E*: Scott.Slattery@MotorolaSolutions.com
>
>
>
>
> On Wed, Mar 24, 2021 at 2:14 PM David Lang <david@lang.hm> wrote:
>
> > when you say 'each is incorrect but the same format'
> >
> > does that mean that every log has the year missing? or that every log is
> > combining the logs together?
> >
> > I'll note that it's possible to define a custom parser using the
> > mmnormalize
> > library and add it to the parsing stack. I helped define the need for such
> > a
> > capability, but haven't used it yet.
> >
> > David Lang
> >
> > On Wed, 24 Mar 2021, Scott Slattery wrote:
> >
> > > Date: Wed, 24 Mar 2021 08:58:07 -0700
> > > From: Scott Slattery <scott.slattery@motorolasolutions.com>
> > > To: mariusz.kruk@safecomp.com
> > > Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <david@lang.hm
> > >
> > > Subject: Re: [rsyslog] Altering forwarded logfile names
> > >
> > > Yes, all the logs are coming in this way. There are three separate logs
> > > being sent and each is incorrect in its own right but the same format
> > > behavior is consistent across each of the three logs. I will look further
> > > into the suggestions you have provided and have already reached out to
> > the
> > > appliance support team and will raise the nonconformance with the RFC.
> > >
> > > This does provide me some valuable guidance and I thank each of you for
> > > your feedback.
> > >
> > > *Scott Slattery*
> > >
> > > *Sr. Systems & Cloud Architect*
> > >
> > > *Cloud, Compute, Information & Architecture Team*
> > >
> > > motorolasolutions.com
> > >
> > > *O: 602.529.8226*
> > >
> > > *E*: Scott.Slattery@MotorolaSolutions.com
> > >
> > >
> > >
> > >
> > > On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com> wrote:
> > >
> > >> There's something strange with this log entry. Notice that it looks as
> > if
> > >> it were more than one log entries joined with LF character (escaped on
> > >> input as #012). And all "further" entries start with a - still
> > >> nonconformant to the RFC but "fuller" - timestamp containing year.It's
> > the
> > >> leading event that seems to be trimmed in the front somehow. Do all
> > events
> > >> arrive like this?Wys?ano z telefonu Samsung<div>
> > >> </div><div>
> > >> </div><!-- originalMessage --><div>-------- Original message
> > >> --------</div><div>From: Scott Slattery via rsyslog <
> > >> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021 01:52
> > >> (GMT+01:00) </div><div>To: David Lang <david@lang.hm> </div><div>Cc:
> > >> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott Slattery
> > via
> > >> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re: [rsyslog]
> > >> Altering forwarded logfile names </div><div>
> > >> </div>
> > >>
> > >> Hi David, fortunately I had already done this. I'm including an actual
> > log
> > >> entry but have anonymized the data to keep the actual user and email
> > >> address confidential:
> > >>
> > >> Debug line with all properties:
> > >> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
> > >> , fromhost-ip: '10.41.102.143'
> > >> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
> > >> , PRI: 13,
> > >> syslogtag '03-23T16:'
> > >> , programname: '03-23T16'
> > >> , APP-NAME: '03-23T16'
> > >> , PROCID: '-'
> > >> , MSGID: '-'
> > >> , TIMNESTAMP: 'Mar 23 21:47:13'
> > >> , STRUCTURED-DATA: '-'
> > >> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> > ACCESS_GATEWAY
> > >> ACCESS AUTHZ SESSION INFO USER_SESSION
> > >> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> > >> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> > >> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> > >> apigniodashboard-uat.mycompany.com"
> > >> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> > REMOTE_IP="10.44.65.38"
> > >> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> > >> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> > Safari/537.36"]
> > >> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> > >> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> > >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
> > INFO
> > >> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> > >> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> > >> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> > APP="Ignio
> > >> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> > >> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> > >> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> > >> Joe.User@mycompantions.com RelayDomain=
> > apigniodashboard-uat.mycompany.com
> > >> UserName=Joe.User@mycompantions.com
> > >>
> > >>
> > SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> > >> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
> > >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> > >> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> > >> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> > >> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac
> > OS X
> > >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> > >> Safari/537.36"] allow access to
> > resource#0122021-03-23T16:47:20.711-05:00
> > >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION
> > INFO
> > >> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> > >> SUBJECT="Joe.User@mycompan'
> > >> *escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> > >> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
> > >> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> > >> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> > >> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> > >> apigniodashboard-uat.mycompany.com"
> > >> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> > REMOTE_IP="10.44.65.38"
> > >> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> > >> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> > Safari/537.36"]
> > >> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> > >> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> > >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
> > INFO
> > >> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> > >> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> > >> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> > APP="Ignio
> > >> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> > >> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> > >> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> > >> Joe.User@mycompantions.com RelayDomain=
> > apigniodashboard-uat.mycompany.com
> > >> UserName=Joe.User@mycompantions.com
> > >>
> > >>
> > SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> > >> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
> > >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> > >> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> > >> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> > >> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac
> > OS X
> > >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> > >> Safari/537.36"] allow access to
> > resource#0122021-03-23T16:47:20.711-05:00
> > >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION
> > INFO
> > >> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> > >> SUBJECT="Joe.User@mycompan'
> > >> *rawmsg*: '03-23T16:47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> > >> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
> > >> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> > >> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> > >> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> > >> apigniodashboard-uat.mycompany.com"
> > >> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> > REMOTE_IP="10.44.65.38"
> > >> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> > >> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> > Safari/537.36"]
> > >> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> > >> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> > >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
> > INFO
> > >> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> > >> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> > >> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> > APP="Ignio
> > >> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> > >> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> > >> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> > >> Joe.User@mycompantions.com RelayDomain=
> > apigniodashboard-uat.mycompany.com
> > >> UserName=Joe.User@mycompantions.com
> > >>
> > >>
> > SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> > >> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
> > >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> > >> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> > >> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> > >> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac
> > OS X
> > >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> > >> Safari/537.36"] allow access to
> > resource#0122021-03-23T16:47:20.711-05:00
> > >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION
> > INFO
> > >> USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> > >> SUBJECT="Joe.User@mycompan'
> > >>
> > >>
> > >> By inspecting the rawmsg, I can see that field four (space-delimited)
> > >> indicates this is the ACCESS log. So if I were able to extract the log
> > >> identifier from the msg, I could then write all access logs to the same
> > >> daily file. There are other formats as well from the same device but the
> > >> idea is the same.
> > >>
> > >> *Scott Slattery*
> > >>
> > >> *Sr. Systems & Cloud Architect*
> > >>
> > >> *Cloud, Compute, Information & Architecture Team*
> > >>
> > >> motorolasolutions.com
> > >>
> > >> *O: 602.529.8226*
> > >>
> > >> *E*: Scott.Slattery@MotorolaSolutions.com
> > >>
> > >>
> > >>
> > >>
> > >> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm> wrote:
> > >>
> > >>> the source logfile name is not included in the payload by the syslog
> > >> spec.
> > >>> It
> > >>> may be in the case of your appliance, but we would need to see a sample
> > >>> log to
> > >>> understand ho to parse it.
> > >>>
> > >>> based on your template, you are using app-name, which may be listed
> > >>> separtely if
> > >>> it's a RFC5424 format log, or may be part of the syslog tag if it's a
> > >>> RFC3164
> > >>> format log over the wire (neither format has a way to specify a source
> > >> log
> > >>> file
> > >>> by default)
> > >>>
> > >>> you can look at
> > >>>
> > >>
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
> > >>> and see the *-cc
> > >>> options that you could apply to the app-name to eliminate control
> > >>> characters.
> > >>>
> > >>> Again, we really need to see the original log message to understand
> > >> what's
> > >>> what.
> > >>> Please log it with the templateRSYSLOG_DebugFormat so we can see
> > exactly
> > >>> what is
> > >>> sent over the wire and how rsyslog has parsed it.
> > >>>
> > >>> David Lang
> > >>>
> > >>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
> > >>> wrote:
> > >>>
> > >>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
> > >>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
> > >>>> To: John Chivian <jchivian@chivian.com>
> > >>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
> > >>>> rsyslog-users <rsyslog@lists.adiscon.com>
> > >>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> > >>>>
> > >>>> Thanks, John, let me try to clarify what I mean.
> > >>>>
> > >>>> Normally when I forward from a remote server to the central log
> > >> server, I
> > >>>> can include a tag that can then be used to determine the file name I
> > >> want
> > >>>> on the central server. Since I have no real way to include this tag
> > >> from
> > >>>> the appliance, this is not an option.
> > >>>>
> > >>>> I'm looking for a way of inspecting the incoming packets to
> > determining
> > >>> the
> > >>>> source logfile name (which is included in the payload) and use that
> > >>>> filename on the target central server. Since there are multiple logs
> > >>> being
> > >>>> sent (access, audit, monitor, etc.), I'd like to segregate these into
> > >>> their
> > >>>> own files. I'm already using a template with the host information to
> > >>>> dynamically create the file names. I just don't know how I can go
> > >> beyond
> > >>>> this to also include the source logname.
> > >>>>
> > >>>> Here's the template I'm using. It works for all other hosts where I
> > can
> > >>>> configure the tag but I get garbage names from the appliance. I had
> > >> hoped
> > >>>> that the appliance included some standard syslog tags but it doesn't
> > >> seem
> > >>>> so.
> > >>>>
> > >>>> template(name="DynRemoteLogFile" type="string"
> > >>>>
> > >>>
> > >>
> > string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
> > >>>>
> > >>>> *Scott Slattery*
> > >>>>
> > >>>> *Sr. Systems & Cloud Architect*
> > >>>>
> > >>>> *Cloud, Compute, Information & Architecture Team*
> > >>>>
> > >>>> motorolasolutions.com
> > >>>>
> > >>>> *O: 602.529.8226*
> > >>>>
> > >>>> *E*: Scott.Slattery@MotorolaSolutions.com
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <jchivian@chivian.com>
> > >>> wrote:
> > >>>>
> > >>>>> Your use of the term “file name” is confusing. When senders deliver
> > >> to
> > >>>>> rsyslog over the network there is no exchange of files or filenames,
> > >>> only
> > >>>>> packets of information. Those packets are expected to be in a format
> > >>> that
> > >>>>> syslog understands such that useful information (header elements and
> > >>>>> message body) may be parsed from them. If you as the rsyslog admin
> > >>> choose
> > >>>>> to use some of that header information to compose filenames for
> > output
> > >>>>> files, then yes you are sort of at the mercy of the senders content
> > >>>>> (especially if the sender doesn’t follow the syslog rules). However,
> > >>> there
> > >>>>> are functions in the advanced syntax that can be used to perform the
> > >>> type
> > >>>>> of character replacements you’re talking about.
> > >>>>>
> > >>>>> It is common practice to use the syslog header/rsyslog property
> > >> element
> > >>>>> called “hostname” for just such purposes. Is this what you’re
> > talking
> > >>>>> about? You’d have to provide your configuration for real analysis,
> > at
> > >>>>> least the part you perceive to be responsible for the problem.
> > >>>>>
> > >>>>> Regards,
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
> > >>>>> rsyslog@lists.adiscon.com> wrote:
> > >>>>>>
> > >>>>>> I have a configured central log collector using rsyslog. A few of
> > >> the
> > >>>>>> devices forwarding their logs are appliances that have no
> > >>> configuration
> > >>>>>> options other than the IP forwarding address and protocol. I cannot
> > >>>>> control
> > >>>>>> what file names are being sent.
> > >>>>>>
> > >>>>>> Unfortunately, they are sending unintelligible file names with
> > >>> characters
> > >>>>>> that normally would be escaped. Is there any way I can control or
> > >>> alter
> > >>>>> the
> > >>>>>> incoming file name to normalize it to avoid these odd characters?
> > >>>>>>
> > >>>>>> For example, could I establish a character map that maps the
> > >> unallowed
> > >>>>>> character to something acceptable?
> > >>>>>>
> > >>>>>> thanks,
> > >>>>>>
> > >>>>>> *Scott Slattery*
> > >>>>>>
> > >>>>>> *Sr. Systems & Cloud Architect*
> > >>>>>>
> > >>>>>> *Cloud, Compute, Information & Architecture Team*
> > >>>>>>
> > >>>>>> motorolasolutions.com
> > >>>>>>
> > >>>>>> *O: 602.529.822*
> > >>>>>>
> > >>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> > >>>>>>
> > >>>>>> --
> > >>>>>>
> > >>>>>>
> > >>>>>> *For more information on how and why we collect your personal
> > >>>>>> information, please visit our Privacy Policy
> > >>>>>> <
> > >>>>>
> > >>>
> > >>
> > https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> > >>>>>> .*
> > >>>>>> _______________________________________________
> > >>>>>> rsyslog mailing list
> > >>>>>>
> > >>>>>
> > >>>
> > >>
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
> > >>>>>>
> > >>>>>
> > >>>
> > >>
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
> > >>>>>> What's up with rsyslog? Follow
> > >>>>>
> > >>>
> > >>
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
> > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > >>> myriad
> > >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> > you
> > >>>>> DON'T LIKE THAT.
> > >>>>>
> > >>>>>
> > >>>>
> > >>>> --
> > >>>>
> > >>>>
> > >>>> *For more information on how and why we collect your personal
> > >>>> information, please visit our Privacy Policy
> > >>>> <
> > >>>
> > >>
> > https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> > >>>> .*
> > >>>> _______________________________________________
> > >>>> rsyslog mailing list
> > >>>>
> > >>>
> > >>
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
> > >>>>
> > >>>
> > >>
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
> > >>>> What's up with rsyslog? Follow
> > >>>
> > >>
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
> > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > >> myriad
> > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > >>> DON'T LIKE THAT.
> > >>
> > >> --
> > >>
> > >>
> > >> *For more information on how and why we collect your personal
> > >> information, please visit our Privacy Policy
> > >> <
> > >>
> > https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> > >>> .*
> > >> _______________________________________________
> > >> rsyslog mailing list
> > >>
> > >>
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
> > >>
> > >>
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
> > >> What's up with rsyslog? Follow
> > >>
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
> > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > >> DON'T LIKE THAT.
> > >
> > >
>
> --
>
>
> *For more information on how and why we collect your personal
> information, please visit our Privacy Policy
> <https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
Hey Guys, before I consume more of your time, I'm capturing the packets
from the source device to determine if any of the payload is lost on its
way to the rsyslog collector. After I analyze further, I'll share my
observations. I'm hoping nothing is getting dropped over the wire. Thanks
for your patience and help so far. I expect to have additional details
shortly.

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: Scott.Slattery@MotorolaSolutions.com




On Thu, Mar 25, 2021 at 12:51 AM Rainer Gerhards <rgerhards@hq.adiscon.com>
wrote:

> Sorry if I didn't catch it, but did you post the configuration? If
> not, it would probably be useful.
>
> Looking at the message properties, this looks like the message was
> received via UDP but had a LF inside it which was intended as a frame
> separator. If so, that is another bug in the vendor's implementation.
> Framing is used for TCP, but not UDP. For UDP it is strictly one
> message per package.
>
> HTH,
> Rainer
>
> El mié, 24 mar 2021 a las 22:23, Scott Slattery via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
> >
> > From what I can discern since there are lots of transactions, there seem
> to
> > be multiple payload formats. For all that I have so far analyzed, the
> date
> > truncation issue is the same regardless of the payload received. Like
> you,
> > I've never used mmnormalize but I will look at it as an option. I've
> > reached out to the product support team to better understand if there is
> a
> > appliance-side misconfiguration and something that can be corrected there
> > or if I need to make adjustments on the rsyslog side to clean up the
> > records.
> >
> > What I meant is that there are multiple message formats (payload data)
> but
> > there seems to be a consistent truncation of the date field just as you
> saw
> > in the app-name tag field.
> >
> > I appreciate your ongoing feedback and help. This issue is new to me and
> > will need a creative solution if the vendor has no answers.
> >
> > Many thanks,
> >
> >
> > *Scott Slattery*
> >
> > *Sr. Systems & Cloud Architect*
> >
> > *Cloud, Compute, Information & Architecture Team*
> >
> > motorolasolutions.com
> >
> > *O: 602.529.8226*
> >
> > *E*: Scott.Slattery@MotorolaSolutions.com
> >
> >
> >
> >
> > On Wed, Mar 24, 2021 at 2:14 PM David Lang <david@lang.hm> wrote:
> >
> > > when you say 'each is incorrect but the same format'
> > >
> > > does that mean that every log has the year missing? or that every log
> is
> > > combining the logs together?
> > >
> > > I'll note that it's possible to define a custom parser using the
> > > mmnormalize
> > > library and add it to the parsing stack. I helped define the need for
> such
> > > a
> > > capability, but haven't used it yet.
> > >
> > > David Lang
> > >
> > > On Wed, 24 Mar 2021, Scott Slattery wrote:
> > >
> > > > Date: Wed, 24 Mar 2021 08:58:07 -0700
> > > > From: Scott Slattery <scott.slattery@motorolasolutions.com>
> > > > To: mariusz.kruk@safecomp.com
> > > > Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
> david@lang.hm
> > > >
> > > > Subject: Re: [rsyslog] Altering forwarded logfile names
> > > >
> > > > Yes, all the logs are coming in this way. There are three separate
> logs
> > > > being sent and each is incorrect in its own right but the same format
> > > > behavior is consistent across each of the three logs. I will look
> further
> > > > into the suggestions you have provided and have already reached out
> to
> > > the
> > > > appliance support team and will raise the nonconformance with the
> RFC.
> > > >
> > > > This does provide me some valuable guidance and I thank each of you
> for
> > > > your feedback.
> > > >
> > > > *Scott Slattery*
> > > >
> > > > *Sr. Systems & Cloud Architect*
> > > >
> > > > *Cloud, Compute, Information & Architecture Team*
> > > >
> > > > motorolasolutions.com
> > > >
> > > > *O: 602.529.8226*
> > > >
> > > > *E*: Scott.Slattery@MotorolaSolutions.com
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com> wrote:
> > > >
> > > >> There's something strange with this log entry. Notice that it looks
> as
> > > if
> > > >> it were more than one log entries joined with LF character (escaped
> on
> > > >> input as #012). And all "further" entries start with a - still
> > > >> nonconformant to the RFC but "fuller" - timestamp containing
> year.It's
> > > the
> > > >> leading event that seems to be trimmed in the front somehow. Do all
> > > events
> > > >> arrive like this?Wys?ano z telefonu Samsung<div>
> > > >> </div><div>
> > > >> </div><!-- originalMessage --><div>-------- Original message
> > > >> --------</div><div>From: Scott Slattery via rsyslog <
> > > >> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021 01:52
> > > >> (GMT+01:00) </div><div>To: David Lang <david@lang.hm>
> </div><div>Cc:
> > > >> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott
> Slattery
> > > via
> > > >> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re:
> [rsyslog]
> > > >> Altering forwarded logfile names </div><div>
> > > >> </div>
> > > >>
> > > >> Hi David, fortunately I had already done this. I'm including an
> actual
> > > log
> > > >> entry but have anonymized the data to keep the actual user and email
> > > >> address confidential:
> > > >>
> > > >> Debug line with all properties:
> > > >> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
> > > >> , fromhost-ip: '10.41.102.143'
> > > >> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
> > > >> , PRI: 13,
> > > >> syslogtag '03-23T16:'
> > > >> , programname: '03-23T16'
> > > >> , APP-NAME: '03-23T16'
> > > >> , PROCID: '-'
> > > >> , MSGID: '-'
> > > >> , TIMNESTAMP: 'Mar 23 21:47:13'
> > > >> , STRUCTURED-DATA: '-'
> > > >> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> > > ACCESS_GATEWAY
> > > >> ACCESS AUTHZ SESSION INFO USER_SESSION
> > > >> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> > > >> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> > > >> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> > > >> apigniodashboard-uat.mycompany.com"
> > > >> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> > > REMOTE_IP="10.44.65.38"
> > > >> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> > > >> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> > > Safari/537.36"]
> > > >> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> > > >> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> > > >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
> > > INFO
> > > >> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> > > >> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> > > >> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> > > APP="Ignio
> > > >> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> > > >> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> > > >> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> > > >> Joe.User@mycompantions.com RelayDomain=
> > > apigniodashboard-uat.mycompany.com
> > > >> UserName=Joe.User@mycompantions.com
> > > >>
> > > >>
> > >
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> > > >> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
> OS X
> > > >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> > > >> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> > > >> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> > > >> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
> Mac
> > > OS X
> > > >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> > > >> Safari/537.36"] allow access to
> > > resource#0122021-03-23T16:47:20.711-05:00
> > > >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> SESSION
> > > INFO
> > > >> USER_SESSION
> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> > > >> SUBJECT="Joe.User@mycompan'
> > > >> *escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> > > >> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
> > > >> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> > > >> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> > > >> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> > > >> apigniodashboard-uat.mycompany.com"
> > > >> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> > > REMOTE_IP="10.44.65.38"
> > > >> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> > > >> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> > > Safari/537.36"]
> > > >> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> > > >> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> > > >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
> > > INFO
> > > >> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> > > >> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> > > >> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> > > APP="Ignio
> > > >> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> > > >> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> > > >> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> > > >> Joe.User@mycompantions.com RelayDomain=
> > > apigniodashboard-uat.mycompany.com
> > > >> UserName=Joe.User@mycompantions.com
> > > >>
> > > >>
> > >
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> > > >> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
> OS X
> > > >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> > > >> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> > > >> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> > > >> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
> Mac
> > > OS X
> > > >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> > > >> Safari/537.36"] allow access to
> > > resource#0122021-03-23T16:47:20.711-05:00
> > > >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> SESSION
> > > INFO
> > > >> USER_SESSION
> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> > > >> SUBJECT="Joe.User@mycompan'
> > > >> *rawmsg*: '03-23T16:47:20.708-05:00
> AUSE1OAGATST02.aws.mycompany.com
> > > >> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
> > > >> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> > > >> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> > > >> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> > > >> apigniodashboard-uat.mycompany.com"
> > > >> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> > > REMOTE_IP="10.44.65.38"
> > > >> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> > > >> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> > > Safari/537.36"]
> > > >> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> > > >> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> > > >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
> > > INFO
> > > >> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> > > >> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> > > >> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> > > APP="Ignio
> > > >> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> > > >> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> > > >> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> > > >> Joe.User@mycompantions.com RelayDomain=
> > > apigniodashboard-uat.mycompany.com
> > > >> UserName=Joe.User@mycompantions.com
> > > >>
> > > >>
> > >
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> > > >> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
> OS X
> > > >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> > > >> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> > > >> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> > > >> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
> Mac
> > > OS X
> > > >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> > > >> Safari/537.36"] allow access to
> > > resource#0122021-03-23T16:47:20.711-05:00
> > > >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> SESSION
> > > INFO
> > > >> USER_SESSION
> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> > > >> SUBJECT="Joe.User@mycompan'
> > > >>
> > > >>
> > > >> By inspecting the rawmsg, I can see that field four
> (space-delimited)
> > > >> indicates this is the ACCESS log. So if I were able to extract the
> log
> > > >> identifier from the msg, I could then write all access logs to the
> same
> > > >> daily file. There are other formats as well from the same device
> but the
> > > >> idea is the same.
> > > >>
> > > >> *Scott Slattery*
> > > >>
> > > >> *Sr. Systems & Cloud Architect*
> > > >>
> > > >> *Cloud, Compute, Information & Architecture Team*
> > > >>
> > > >> motorolasolutions.com
> > > >>
> > > >> *O: 602.529.8226*
> > > >>
> > > >> *E*: Scott.Slattery@MotorolaSolutions.com
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm> wrote:
> > > >>
> > > >>> the source logfile name is not included in the payload by the
> syslog
> > > >> spec.
> > > >>> It
> > > >>> may be in the case of your appliance, but we would need to see a
> sample
> > > >>> log to
> > > >>> understand ho to parse it.
> > > >>>
> > > >>> based on your template, you are using app-name, which may be listed
> > > >>> separtely if
> > > >>> it's a RFC5424 format log, or may be part of the syslog tag if
> it's a
> > > >>> RFC3164
> > > >>> format log over the wire (neither format has a way to specify a
> source
> > > >> log
> > > >>> file
> > > >>> by default)
> > > >>>
> > > >>> you can look at
> > > >>>
> > > >>
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
> > > >>> and see the *-cc
> > > >>> options that you could apply to the app-name to eliminate control
> > > >>> characters.
> > > >>>
> > > >>> Again, we really need to see the original log message to understand
> > > >> what's
> > > >>> what.
> > > >>> Please log it with the templateRSYSLOG_DebugFormat so we can see
> > > exactly
> > > >>> what is
> > > >>> sent over the wire and how rsyslog has parsed it.
> > > >>>
> > > >>> David Lang
> > > >>>
> > > >>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
> > > >>> wrote:
> > > >>>
> > > >>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
> > > >>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
> > > >>>> To: John Chivian <jchivian@chivian.com>
> > > >>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
> > > >>>> rsyslog-users <rsyslog@lists.adiscon.com>
> > > >>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> > > >>>>
> > > >>>> Thanks, John, let me try to clarify what I mean.
> > > >>>>
> > > >>>> Normally when I forward from a remote server to the central log
> > > >> server, I
> > > >>>> can include a tag that can then be used to determine the file
> name I
> > > >> want
> > > >>>> on the central server. Since I have no real way to include this
> tag
> > > >> from
> > > >>>> the appliance, this is not an option.
> > > >>>>
> > > >>>> I'm looking for a way of inspecting the incoming packets to
> > > determining
> > > >>> the
> > > >>>> source logfile name (which is included in the payload) and use
> that
> > > >>>> filename on the target central server. Since there are multiple
> logs
> > > >>> being
> > > >>>> sent (access, audit, monitor, etc.), I'd like to segregate these
> into
> > > >>> their
> > > >>>> own files. I'm already using a template with the host information
> to
> > > >>>> dynamically create the file names. I just don't know how I can go
> > > >> beyond
> > > >>>> this to also include the source logname.
> > > >>>>
> > > >>>> Here's the template I'm using. It works for all other hosts where
> I
> > > can
> > > >>>> configure the tag but I get garbage names from the appliance. I
> had
> > > >> hoped
> > > >>>> that the appliance included some standard syslog tags but it
> doesn't
> > > >> seem
> > > >>>> so.
> > > >>>>
> > > >>>> template(name="DynRemoteLogFile" type="string"
> > > >>>>
> > > >>>
> > > >>
> > >
> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
> > > >>>>
> > > >>>> *Scott Slattery*
> > > >>>>
> > > >>>> *Sr. Systems & Cloud Architect*
> > > >>>>
> > > >>>> *Cloud, Compute, Information & Architecture Team*
> > > >>>>
> > > >>>> motorolasolutions.com
> > > >>>>
> > > >>>> *O: 602.529.8226*
> > > >>>>
> > > >>>> *E*: Scott.Slattery@MotorolaSolutions.com
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <
> jchivian@chivian.com>
> > > >>> wrote:
> > > >>>>
> > > >>>>> Your use of the term “file name” is confusing. When senders
> deliver
> > > >> to
> > > >>>>> rsyslog over the network there is no exchange of files or
> filenames,
> > > >>> only
> > > >>>>> packets of information. Those packets are expected to be in a
> format
> > > >>> that
> > > >>>>> syslog understands such that useful information (header elements
> and
> > > >>>>> message body) may be parsed from them. If you as the rsyslog
> admin
> > > >>> choose
> > > >>>>> to use some of that header information to compose filenames for
> > > output
> > > >>>>> files, then yes you are sort of at the mercy of the senders
> content
> > > >>>>> (especially if the sender doesn’t follow the syslog rules).
> However,
> > > >>> there
> > > >>>>> are functions in the advanced syntax that can be used to perform
> the
> > > >>> type
> > > >>>>> of character replacements you’re talking about.
> > > >>>>>
> > > >>>>> It is common practice to use the syslog header/rsyslog property
> > > >> element
> > > >>>>> called “hostname” for just such purposes. Is this what you’re
> > > talking
> > > >>>>> about? You’d have to provide your configuration for real
> analysis,
> > > at
> > > >>>>> least the part you perceive to be responsible for the problem.
> > > >>>>>
> > > >>>>> Regards,
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
> > > >>>>> rsyslog@lists.adiscon.com> wrote:
> > > >>>>>>
> > > >>>>>> I have a configured central log collector using rsyslog. A few
> of
> > > >> the
> > > >>>>>> devices forwarding their logs are appliances that have no
> > > >>> configuration
> > > >>>>>> options other than the IP forwarding address and protocol. I
> cannot
> > > >>>>> control
> > > >>>>>> what file names are being sent.
> > > >>>>>>
> > > >>>>>> Unfortunately, they are sending unintelligible file names with
> > > >>> characters
> > > >>>>>> that normally would be escaped. Is there any way I can control
> or
> > > >>> alter
> > > >>>>> the
> > > >>>>>> incoming file name to normalize it to avoid these odd
> characters?
> > > >>>>>>
> > > >>>>>> For example, could I establish a character map that maps the
> > > >> unallowed
> > > >>>>>> character to something acceptable?
> > > >>>>>>
> > > >>>>>> thanks,
> > > >>>>>>
> > > >>>>>> *Scott Slattery*
> > > >>>>>>
> > > >>>>>> *Sr. Systems & Cloud Architect*
> > > >>>>>>
> > > >>>>>> *Cloud, Compute, Information & Architecture Team*
> > > >>>>>>
> > > >>>>>> motorolasolutions.com
> > > >>>>>>
> > > >>>>>> *O: 602.529.822*
> > > >>>>>>
> > > >>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> *For more information on how and why we collect your personal
> > > >>>>>> information, please visit our Privacy Policy
> > > >>>>>> <
> > > >>>>>
> > > >>>
> > > >>
> > >
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> > > >>>>>> .*
> > > >>>>>> _______________________________________________
> > > >>>>>> rsyslog mailing list
> > > >>>>>>
> > > >>>>>
> > > >>>
> > > >>
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
> > > >>>>>>
> > > >>>>>
> > > >>>
> > > >>
> > >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
> > > >>>>>> What's up with rsyslog? Follow
> > > >>>>>
> > > >>>
> > > >>
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
> > > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
> a
> > > >>> myriad
> > > >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if
> > > you
> > > >>>>> DON'T LIKE THAT.
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>> --
> > > >>>>
> > > >>>>
> > > >>>> *For more information on how and why we collect your personal
> > > >>>> information, please visit our Privacy Policy
> > > >>>> <
> > > >>>
> > > >>
> > >
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> > > >>>> .*
> > > >>>> _______________________________________________
> > > >>>> rsyslog mailing list
> > > >>>>
> > > >>>
> > > >>
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
> > > >>>>
> > > >>>
> > > >>
> > >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
> > > >>>> What's up with rsyslog? Follow
> > > >>>
> > > >>
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
> > > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > > >> myriad
> > > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> you
> > > >>> DON'T LIKE THAT.
> > > >>
> > > >> --
> > > >>
> > > >>
> > > >> *For more information on how and why we collect your personal
> > > >> information, please visit our Privacy Policy
> > > >> <
> > > >>
> > >
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> > > >>> .*
> > > >> _______________________________________________
> > > >> rsyslog mailing list
> > > >>
> > > >>
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
> > > >>
> > > >>
> > >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
> > > >> What's up with rsyslog? Follow
> > > >>
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
> > > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> you
> > > >> DON'T LIKE THAT.
> > > >
> > > >
> >
> > --
> >
> >
> > *For more information on how and why we collect your personal
> > information, please visit our Privacy Policy
> > <
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >.*
> > _______________________________________________
> > rsyslog mailing list
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=7hc9b2gSfvsfQzXvsRVXqGeP4Rz_tJtrykpjqkGBIdk&e=
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=bDybhvts9KqI-WEfMHo7O40ulQK5M6Tg4tnKNRRIhE8&e=
> > What's up with rsyslog? Follow
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=zPhpcUPlK_CWLiU75_WwjyQGz5mFTiXc1yyBIfppkrI&e=
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>

--


*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
the rawmsg field in the debugformat output shows exactly what rsyslog is seeing.

the reason I asked you to check multiple entries is that if rsyslog does not see
the separator (due to either multiple messages in one UDP packet, or missing
newlines in a TCP stream) it will combine what are intended as multiple messages
together into one message as far as rsyslog is concerned, and only break it into
multiple messages when it hits the configured maxmessagesize size. That breaks
things at arbitrary points in the message, so as you look at multiple messages,
they will not all start the same way.

a tcpdump -s0 -A -n (with the appropriate port/ip filter) will also show exactly
what's happening over the wire.

David Lang

On Thu, 25 Mar 2021, Scott Slattery wrote:

> Date: Thu, 25 Mar 2021 12:23:03 -0700
> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> To: Rainer Gerhards <rgerhards@hq.adiscon.com>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <david@lang.hm>,
> mariusz.kruk@safecomp.com
> Subject: Re: [rsyslog] Altering forwarded logfile names
>
> Hey Guys, before I consume more of your time, I'm capturing the packets
> from the source device to determine if any of the payload is lost on its
> way to the rsyslog collector. After I analyze further, I'll share my
> observations. I'm hoping nothing is getting dropped over the wire. Thanks
> for your patience and help so far. I expect to have additional details
> shortly.
>
> *Scott Slattery*
>
> *Sr. Systems & Cloud Architect*
>
> *Cloud, Compute, Information & Architecture Team*
>
> motorolasolutions.com
>
> *O: 602.529.8226*
>
> *E*: Scott.Slattery@MotorolaSolutions.com
>
>
>
>
> On Thu, Mar 25, 2021 at 12:51 AM Rainer Gerhards <rgerhards@hq.adiscon.com>
> wrote:
>
>> Sorry if I didn't catch it, but did you post the configuration? If
>> not, it would probably be useful.
>>
>> Looking at the message properties, this looks like the message was
>> received via UDP but had a LF inside it which was intended as a frame
>> separator. If so, that is another bug in the vendor's implementation.
>> Framing is used for TCP, but not UDP. For UDP it is strictly one
>> message per package.
>>
>> HTH,
>> Rainer
>>
>> El mié, 24 mar 2021 a las 22:23, Scott Slattery via rsyslog
>> (<rsyslog@lists.adiscon.com>) escribió:
>>>
>>> From what I can discern since there are lots of transactions, there seem
>> to
>>> be multiple payload formats. For all that I have so far analyzed, the
>> date
>>> truncation issue is the same regardless of the payload received. Like
>> you,
>>> I've never used mmnormalize but I will look at it as an option. I've
>>> reached out to the product support team to better understand if there is
>> a
>>> appliance-side misconfiguration and something that can be corrected there
>>> or if I need to make adjustments on the rsyslog side to clean up the
>>> records.
>>>
>>> What I meant is that there are multiple message formats (payload data)
>> but
>>> there seems to be a consistent truncation of the date field just as you
>> saw
>>> in the app-name tag field.
>>>
>>> I appreciate your ongoing feedback and help. This issue is new to me and
>>> will need a creative solution if the vendor has no answers.
>>>
>>> Many thanks,
>>>
>>>
>>> *Scott Slattery*
>>>
>>> *Sr. Systems & Cloud Architect*
>>>
>>> *Cloud, Compute, Information & Architecture Team*
>>>
>>> motorolasolutions.com
>>>
>>> *O: 602.529.8226*
>>>
>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>
>>>
>>>
>>>
>>> On Wed, Mar 24, 2021 at 2:14 PM David Lang <david@lang.hm> wrote:
>>>
>>>> when you say 'each is incorrect but the same format'
>>>>
>>>> does that mean that every log has the year missing? or that every log
>> is
>>>> combining the logs together?
>>>>
>>>> I'll note that it's possible to define a custom parser using the
>>>> mmnormalize
>>>> library and add it to the parsing stack. I helped define the need for
>> such
>>>> a
>>>> capability, but haven't used it yet.
>>>>
>>>> David Lang
>>>>
>>>> On Wed, 24 Mar 2021, Scott Slattery wrote:
>>>>
>>>>> Date: Wed, 24 Mar 2021 08:58:07 -0700
>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>>>> To: mariusz.kruk@safecomp.com
>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
>> david@lang.hm
>>>>>
>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>>>
>>>>> Yes, all the logs are coming in this way. There are three separate
>> logs
>>>>> being sent and each is incorrect in its own right but the same format
>>>>> behavior is consistent across each of the three logs. I will look
>> further
>>>>> into the suggestions you have provided and have already reached out
>> to
>>>> the
>>>>> appliance support team and will raise the nonconformance with the
>> RFC.
>>>>>
>>>>> This does provide me some valuable guidance and I thank each of you
>> for
>>>>> your feedback.
>>>>>
>>>>> *Scott Slattery*
>>>>>
>>>>> *Sr. Systems & Cloud Architect*
>>>>>
>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>
>>>>> motorolasolutions.com
>>>>>
>>>>> *O: 602.529.8226*
>>>>>
>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com> wrote:
>>>>>
>>>>>> There's something strange with this log entry. Notice that it looks
>> as
>>>> if
>>>>>> it were more than one log entries joined with LF character (escaped
>> on
>>>>>> input as #012). And all "further" entries start with a - still
>>>>>> nonconformant to the RFC but "fuller" - timestamp containing
>> year.It's
>>>> the
>>>>>> leading event that seems to be trimmed in the front somehow. Do all
>>>> events
>>>>>> arrive like this?Wys?ano z telefonu Samsung<div>
>>>>>> </div><div>
>>>>>> </div><!-- originalMessage --><div>-------- Original message
>>>>>> --------</div><div>From: Scott Slattery via rsyslog <
>>>>>> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021 01:52
>>>>>> (GMT+01:00) </div><div>To: David Lang <david@lang.hm>
>> </div><div>Cc:
>>>>>> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott
>> Slattery
>>>> via
>>>>>> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re:
>> [rsyslog]
>>>>>> Altering forwarded logfile names </div><div>
>>>>>> </div>
>>>>>>
>>>>>> Hi David, fortunately I had already done this. I'm including an
>> actual
>>>> log
>>>>>> entry but have anonymized the data to keep the actual user and email
>>>>>> address confidential:
>>>>>>
>>>>>> Debug line with all properties:
>>>>>> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
>>>>>> , fromhost-ip: '10.41.102.143'
>>>>>> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
>>>>>> , PRI: 13,
>>>>>> syslogtag '03-23T16:'
>>>>>> , programname: '03-23T16'
>>>>>> , APP-NAME: '03-23T16'
>>>>>> , PROCID: '-'
>>>>>> , MSGID: '-'
>>>>>> , TIMNESTAMP: 'Mar 23 21:47:13'
>>>>>> , STRUCTURED-DATA: '-'
>>>>>> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
>>>> ACCESS_GATEWAY
>>>>>> ACCESS AUTHZ SESSION INFO USER_SESSION
>>>>>> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>> apigniodashboard-uat.mycompany.com"
>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>>> REMOTE_IP="10.44.65.38"
>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>> Safari/537.36"]
>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
>>>> INFO
>>>>>> USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
>>>> APP="Ignio
>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
>>>>>> Joe.User@mycompantions.com RelayDomain=
>>>> apigniodashboard-uat.mycompany.com
>>>>>> UserName=Joe.User@mycompantions.com
>>>>>>
>>>>>>
>>>>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
>> OS X
>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
>> Mac
>>>> OS X
>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>> Safari/537.36"] allow access to
>>>> resource#0122021-03-23T16:47:20.711-05:00
>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>> SESSION
>>>> INFO
>>>>>> USER_SESSION
>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>> SUBJECT="Joe.User@mycompan'
>>>>>> *escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
>>>>>> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
>>>>>> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>> apigniodashboard-uat.mycompany.com"
>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>>> REMOTE_IP="10.44.65.38"
>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>> Safari/537.36"]
>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
>>>> INFO
>>>>>> USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
>>>> APP="Ignio
>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
>>>>>> Joe.User@mycompantions.com RelayDomain=
>>>> apigniodashboard-uat.mycompany.com
>>>>>> UserName=Joe.User@mycompantions.com
>>>>>>
>>>>>>
>>>>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
>> OS X
>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
>> Mac
>>>> OS X
>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>> Safari/537.36"] allow access to
>>>> resource#0122021-03-23T16:47:20.711-05:00
>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>> SESSION
>>>> INFO
>>>>>> USER_SESSION
>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>> SUBJECT="Joe.User@mycompan'
>>>>>> *rawmsg*: '03-23T16:47:20.708-05:00
>> AUSE1OAGATST02.aws.mycompany.com
>>>>>> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
>>>>>> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>> apigniodashboard-uat.mycompany.com"
>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>>> REMOTE_IP="10.44.65.38"
>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>> Safari/537.36"]
>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
>>>> INFO
>>>>>> USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
>>>> APP="Ignio
>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
>>>>>> Joe.User@mycompantions.com RelayDomain=
>>>> apigniodashboard-uat.mycompany.com
>>>>>> UserName=Joe.User@mycompantions.com
>>>>>>
>>>>>>
>>>>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
>> OS X
>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
>> Mac
>>>> OS X
>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>> Safari/537.36"] allow access to
>>>> resource#0122021-03-23T16:47:20.711-05:00
>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>> SESSION
>>>> INFO
>>>>>> USER_SESSION
>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>> SUBJECT="Joe.User@mycompan'
>>>>>>
>>>>>>
>>>>>> By inspecting the rawmsg, I can see that field four
>> (space-delimited)
>>>>>> indicates this is the ACCESS log. So if I were able to extract the
>> log
>>>>>> identifier from the msg, I could then write all access logs to the
>> same
>>>>>> daily file. There are other formats as well from the same device
>> but the
>>>>>> idea is the same.
>>>>>>
>>>>>> *Scott Slattery*
>>>>>>
>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>
>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>
>>>>>> motorolasolutions.com
>>>>>>
>>>>>> *O: 602.529.8226*
>>>>>>
>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm> wrote:
>>>>>>
>>>>>>> the source logfile name is not included in the payload by the
>> syslog
>>>>>> spec.
>>>>>>> It
>>>>>>> may be in the case of your appliance, but we would need to see a
>> sample
>>>>>>> log to
>>>>>>> understand ho to parse it.
>>>>>>>
>>>>>>> based on your template, you are using app-name, which may be listed
>>>>>>> separtely if
>>>>>>> it's a RFC5424 format log, or may be part of the syslog tag if
>> it's a
>>>>>>> RFC3164
>>>>>>> format log over the wire (neither format has a way to specify a
>> source
>>>>>> log
>>>>>>> file
>>>>>>> by default)
>>>>>>>
>>>>>>> you can look at
>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
>>>>>>> and see the *-cc
>>>>>>> options that you could apply to the app-name to eliminate control
>>>>>>> characters.
>>>>>>>
>>>>>>> Again, we really need to see the original log message to understand
>>>>>> what's
>>>>>>> what.
>>>>>>> Please log it with the templateRSYSLOG_DebugFormat so we can see
>>>> exactly
>>>>>>> what is
>>>>>>> sent over the wire and how rsyslog has parsed it.
>>>>>>>
>>>>>>> David Lang
>>>>>>>
>>>>>>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
>>>>>>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
>>>>>>>> To: John Chivian <jchivian@chivian.com>
>>>>>>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
>>>>>>>> rsyslog-users <rsyslog@lists.adiscon.com>
>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>>>>>>
>>>>>>>> Thanks, John, let me try to clarify what I mean.
>>>>>>>>
>>>>>>>> Normally when I forward from a remote server to the central log
>>>>>> server, I
>>>>>>>> can include a tag that can then be used to determine the file
>> name I
>>>>>> want
>>>>>>>> on the central server. Since I have no real way to include this
>> tag
>>>>>> from
>>>>>>>> the appliance, this is not an option.
>>>>>>>>
>>>>>>>> I'm looking for a way of inspecting the incoming packets to
>>>> determining
>>>>>>> the
>>>>>>>> source logfile name (which is included in the payload) and use
>> that
>>>>>>>> filename on the target central server. Since there are multiple
>> logs
>>>>>>> being
>>>>>>>> sent (access, audit, monitor, etc.), I'd like to segregate these
>> into
>>>>>>> their
>>>>>>>> own files. I'm already using a template with the host information
>> to
>>>>>>>> dynamically create the file names. I just don't know how I can go
>>>>>> beyond
>>>>>>>> this to also include the source logname.
>>>>>>>>
>>>>>>>> Here's the template I'm using. It works for all other hosts where
>> I
>>>> can
>>>>>>>> configure the tag but I get garbage names from the appliance. I
>> had
>>>>>> hoped
>>>>>>>> that the appliance included some standard syslog tags but it
>> doesn't
>>>>>> seem
>>>>>>>> so.
>>>>>>>>
>>>>>>>> template(name="DynRemoteLogFile" type="string"
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
>>>>>>>>
>>>>>>>> *Scott Slattery*
>>>>>>>>
>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>
>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>
>>>>>>>> motorolasolutions.com
>>>>>>>>
>>>>>>>> *O: 602.529.8226*
>>>>>>>>
>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <
>> jchivian@chivian.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Your use of the term “file name” is confusing. When senders
>> deliver
>>>>>> to
>>>>>>>>> rsyslog over the network there is no exchange of files or
>> filenames,
>>>>>>> only
>>>>>>>>> packets of information. Those packets are expected to be in a
>> format
>>>>>>> that
>>>>>>>>> syslog understands such that useful information (header elements
>> and
>>>>>>>>> message body) may be parsed from them. If you as the rsyslog
>> admin
>>>>>>> choose
>>>>>>>>> to use some of that header information to compose filenames for
>>>> output
>>>>>>>>> files, then yes you are sort of at the mercy of the senders
>> content
>>>>>>>>> (especially if the sender doesn’t follow the syslog rules).
>> However,
>>>>>>> there
>>>>>>>>> are functions in the advanced syntax that can be used to perform
>> the
>>>>>>> type
>>>>>>>>> of character replacements you’re talking about.
>>>>>>>>>
>>>>>>>>> It is common practice to use the syslog header/rsyslog property
>>>>>> element
>>>>>>>>> called “hostname” for just such purposes. Is this what you’re
>>>> talking
>>>>>>>>> about? You’d have to provide your configuration for real
>> analysis,
>>>> at
>>>>>>>>> least the part you perceive to be responsible for the problem.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
>>>>>>>>> rsyslog@lists.adiscon.com> wrote:
>>>>>>>>>>
>>>>>>>>>> I have a configured central log collector using rsyslog. A few
>> of
>>>>>> the
>>>>>>>>>> devices forwarding their logs are appliances that have no
>>>>>>> configuration
>>>>>>>>>> options other than the IP forwarding address and protocol. I
>> cannot
>>>>>>>>> control
>>>>>>>>>> what file names are being sent.
>>>>>>>>>>
>>>>>>>>>> Unfortunately, they are sending unintelligible file names with
>>>>>>> characters
>>>>>>>>>> that normally would be escaped. Is there any way I can control
>> or
>>>>>>> alter
>>>>>>>>> the
>>>>>>>>>> incoming file name to normalize it to avoid these odd
>> characters?
>>>>>>>>>>
>>>>>>>>>> For example, could I establish a character map that maps the
>>>>>> unallowed
>>>>>>>>>> character to something acceptable?
>>>>>>>>>>
>>>>>>>>>> thanks,
>>>>>>>>>>
>>>>>>>>>> *Scott Slattery*
>>>>>>>>>>
>>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>>>
>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>>>
>>>>>>>>>> motorolasolutions.com
>>>>>>>>>>
>>>>>>>>>> *O: 602.529.822*
>>>>>>>>>>
>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *For more information on how and why we collect your personal
>>>>>>>>>> information, please visit our Privacy Policy
>>>>>>>>>> <
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>>>>>> .*
>>>>>>>>>> _______________________________________________
>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
>>>>>>>>>> What's up with rsyslog? Follow
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>> a
>>>>>>> myriad
>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if
>>>> you
>>>>>>>>> DON'T LIKE THAT.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>>
>>>>>>>> *For more information on how and why we collect your personal
>>>>>>>> information, please visit our Privacy Policy
>>>>>>>> <
>>>>>>>
>>>>>>
>>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>>>> .*
>>>>>>>> _______________________________________________
>>>>>>>> rsyslog mailing list
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
>>>>>>>> What's up with rsyslog? Follow
>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>> myriad
>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you
>>>>>>> DON'T LIKE THAT.
>>>>>>
>>>>>> --
>>>>>>
>>>>>>
>>>>>> *For more information on how and why we collect your personal
>>>>>> information, please visit our Privacy Policy
>>>>>> <
>>>>>>
>>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>>> .*
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
>>>>>> What's up with rsyslog? Follow
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad
>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you
>>>>>> DON'T LIKE THAT.
>>>>>
>>>>>
>>>
>>> --
>>>
>>>
>>> *For more information on how and why we collect your personal
>>> information, please visit our Privacy Policy
>>> <
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>> .*
>>> _______________________________________________
>>> rsyslog mailing list
>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=7hc9b2gSfvsfQzXvsRVXqGeP4Rz_tJtrykpjqkGBIdk&e=
>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=bDybhvts9KqI-WEfMHo7O40ulQK5M6Tg4tnKNRRIhE8&e=
>>> What's up with rsyslog? Follow
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=zPhpcUPlK_CWLiU75_WwjyQGz5mFTiXc1yyBIfppkrI&e=
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
Thanks David, I just grabbed my first capture and am looking at it. I'm
using TCP vs UDP but your comment makes sense. I'll certainly share with
you my observations once I have some data to refer to. This has been a
learning experience. It's also the first time I've ever seen such an issue
with rsyslog so it's a bit of a puzzle.

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: Scott.Slattery@MotorolaSolutions.com




On Thu, Mar 25, 2021 at 12:32 PM David Lang <david@lang.hm> wrote:

> the rawmsg field in the debugformat output shows exactly what rsyslog is
> seeing.
>
> the reason I asked you to check multiple entries is that if rsyslog does
> not see
> the separator (due to either multiple messages in one UDP packet, or
> missing
> newlines in a TCP stream) it will combine what are intended as multiple
> messages
> together into one message as far as rsyslog is concerned, and only break
> it into
> multiple messages when it hits the configured maxmessagesize size. That
> breaks
> things at arbitrary points in the message, so as you look at multiple
> messages,
> they will not all start the same way.
>
> a tcpdump -s0 -A -n (with the appropriate port/ip filter) will also show
> exactly
> what's happening over the wire.
>
> David Lang
>
> On Thu, 25 Mar 2021, Scott Slattery wrote:
>
> > Date: Thu, 25 Mar 2021 12:23:03 -0700
> > From: Scott Slattery <scott.slattery@motorolasolutions.com>
> > To: Rainer Gerhards <rgerhards@hq.adiscon.com>
> > Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <david@lang.hm
> >,
> > mariusz.kruk@safecomp.com
> > Subject: Re: [rsyslog] Altering forwarded logfile names
> >
> > Hey Guys, before I consume more of your time, I'm capturing the packets
> > from the source device to determine if any of the payload is lost on its
> > way to the rsyslog collector. After I analyze further, I'll share my
> > observations. I'm hoping nothing is getting dropped over the wire. Thanks
> > for your patience and help so far. I expect to have additional details
> > shortly.
> >
> > *Scott Slattery*
> >
> > *Sr. Systems & Cloud Architect*
> >
> > *Cloud, Compute, Information & Architecture Team*
> >
> > motorolasolutions.com
> >
> > *O: 602.529.8226*
> >
> > *E*: Scott.Slattery@MotorolaSolutions.com
> >
> >
> >
> >
> > On Thu, Mar 25, 2021 at 12:51 AM Rainer Gerhards <
> rgerhards@hq.adiscon.com>
> > wrote:
> >
> >> Sorry if I didn't catch it, but did you post the configuration? If
> >> not, it would probably be useful.
> >>
> >> Looking at the message properties, this looks like the message was
> >> received via UDP but had a LF inside it which was intended as a frame
> >> separator. If so, that is another bug in the vendor's implementation.
> >> Framing is used for TCP, but not UDP. For UDP it is strictly one
> >> message per package.
> >>
> >> HTH,
> >> Rainer
> >>
> >> El mié, 24 mar 2021 a las 22:23, Scott Slattery via rsyslog
> >> (<rsyslog@lists.adiscon.com>) escribió:
> >>>
> >>> From what I can discern since there are lots of transactions, there
> seem
> >> to
> >>> be multiple payload formats. For all that I have so far analyzed, the
> >> date
> >>> truncation issue is the same regardless of the payload received. Like
> >> you,
> >>> I've never used mmnormalize but I will look at it as an option. I've
> >>> reached out to the product support team to better understand if there
> is
> >> a
> >>> appliance-side misconfiguration and something that can be corrected
> there
> >>> or if I need to make adjustments on the rsyslog side to clean up the
> >>> records.
> >>>
> >>> What I meant is that there are multiple message formats (payload data)
> >> but
> >>> there seems to be a consistent truncation of the date field just as you
> >> saw
> >>> in the app-name tag field.
> >>>
> >>> I appreciate your ongoing feedback and help. This issue is new to me
> and
> >>> will need a creative solution if the vendor has no answers.
> >>>
> >>> Many thanks,
> >>>
> >>>
> >>> *Scott Slattery*
> >>>
> >>> *Sr. Systems & Cloud Architect*
> >>>
> >>> *Cloud, Compute, Information & Architecture Team*
> >>>
> >>> motorolasolutions.com
> >>>
> >>> *O: 602.529.8226*
> >>>
> >>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Mar 24, 2021 at 2:14 PM David Lang <david@lang.hm> wrote:
> >>>
> >>>> when you say 'each is incorrect but the same format'
> >>>>
> >>>> does that mean that every log has the year missing? or that every log
> >> is
> >>>> combining the logs together?
> >>>>
> >>>> I'll note that it's possible to define a custom parser using the
> >>>> mmnormalize
> >>>> library and add it to the parsing stack. I helped define the need for
> >> such
> >>>> a
> >>>> capability, but haven't used it yet.
> >>>>
> >>>> David Lang
> >>>>
> >>>> On Wed, 24 Mar 2021, Scott Slattery wrote:
> >>>>
> >>>>> Date: Wed, 24 Mar 2021 08:58:07 -0700
> >>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> >>>>> To: mariusz.kruk@safecomp.com
> >>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
> >> david@lang.hm
> >>>>>
> >>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>>>
> >>>>> Yes, all the logs are coming in this way. There are three separate
> >> logs
> >>>>> being sent and each is incorrect in its own right but the same format
> >>>>> behavior is consistent across each of the three logs. I will look
> >> further
> >>>>> into the suggestions you have provided and have already reached out
> >> to
> >>>> the
> >>>>> appliance support team and will raise the nonconformance with the
> >> RFC.
> >>>>>
> >>>>> This does provide me some valuable guidance and I thank each of you
> >> for
> >>>>> your feedback.
> >>>>>
> >>>>> *Scott Slattery*
> >>>>>
> >>>>> *Sr. Systems & Cloud Architect*
> >>>>>
> >>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>
> >>>>> motorolasolutions.com
> >>>>>
> >>>>> *O: 602.529.8226*
> >>>>>
> >>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com> wrote:
> >>>>>
> >>>>>> There's something strange with this log entry. Notice that it looks
> >> as
> >>>> if
> >>>>>> it were more than one log entries joined with LF character (escaped
> >> on
> >>>>>> input as #012). And all "further" entries start with a - still
> >>>>>> nonconformant to the RFC but "fuller" - timestamp containing
> >> year.It's
> >>>> the
> >>>>>> leading event that seems to be trimmed in the front somehow. Do all
> >>>> events
> >>>>>> arrive like this?Wys?ano z telefonu Samsung<div>
> >>>>>> </div><div>
> >>>>>> </div><!-- originalMessage --><div>-------- Original message
> >>>>>> --------</div><div>From: Scott Slattery via rsyslog <
> >>>>>> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021 01:52
> >>>>>> (GMT+01:00) </div><div>To: David Lang <david@lang.hm>
> >> </div><div>Cc:
> >>>>>> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott
> >> Slattery
> >>>> via
> >>>>>> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re:
> >> [rsyslog]
> >>>>>> Altering forwarded logfile names </div><div>
> >>>>>> </div>
> >>>>>>
> >>>>>> Hi David, fortunately I had already done this. I'm including an
> >> actual
> >>>> log
> >>>>>> entry but have anonymized the data to keep the actual user and email
> >>>>>> address confidential:
> >>>>>>
> >>>>>> Debug line with all properties:
> >>>>>> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
> >>>>>> , fromhost-ip: '10.41.102.143'
> >>>>>> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
> >>>>>> , PRI: 13,
> >>>>>> syslogtag '03-23T16:'
> >>>>>> , programname: '03-23T16'
> >>>>>> , APP-NAME: '03-23T16'
> >>>>>> , PROCID: '-'
> >>>>>> , MSGID: '-'
> >>>>>> , TIMNESTAMP: 'Mar 23 21:47:13'
> >>>>>> , STRUCTURED-DATA: '-'
> >>>>>> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> >>>> ACCESS_GATEWAY
> >>>>>> ACCESS AUTHZ SESSION INFO USER_SESSION
> >>>>>> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> >>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>> apigniodashboard-uat.mycompany.com"
> >>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> >>>> REMOTE_IP="10.44.65.38"
> >>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>> Safari/537.36"]
> >>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
> >>>> INFO
> >>>>>> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> >>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> >>>> APP="Ignio
> >>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> >>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> >>>>>> Joe.User@mycompantions.com RelayDomain=
> >>>> apigniodashboard-uat.mycompany.com
> >>>>>> UserName=Joe.User@mycompantions.com
> >>>>>>
> >>>>>>
> >>>>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
> >> OS X
> >>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> >>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
> >> Mac
> >>>> OS X
> >>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>> Safari/537.36"] allow access to
> >>>> resource#0122021-03-23T16:47:20.711-05:00
> >>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >> SESSION
> >>>> INFO
> >>>>>> USER_SESSION
> >> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>> SUBJECT="Joe.User@mycompan'
> >>>>>> *escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> >>>>>> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
> >>>>>> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> >>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>> apigniodashboard-uat.mycompany.com"
> >>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> >>>> REMOTE_IP="10.44.65.38"
> >>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>> Safari/537.36"]
> >>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
> >>>> INFO
> >>>>>> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> >>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> >>>> APP="Ignio
> >>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> >>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> >>>>>> Joe.User@mycompantions.com RelayDomain=
> >>>> apigniodashboard-uat.mycompany.com
> >>>>>> UserName=Joe.User@mycompantions.com
> >>>>>>
> >>>>>>
> >>>>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
> >> OS X
> >>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> >>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
> >> Mac
> >>>> OS X
> >>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>> Safari/537.36"] allow access to
> >>>> resource#0122021-03-23T16:47:20.711-05:00
> >>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >> SESSION
> >>>> INFO
> >>>>>> USER_SESSION
> >> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>> SUBJECT="Joe.User@mycompan'
> >>>>>> *rawmsg*: '03-23T16:47:20.708-05:00
> >> AUSE1OAGATST02.aws.mycompany.com
> >>>>>> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
> >>>>>> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
> >>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>> apigniodashboard-uat.mycompany.com"
> >>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> >>>> REMOTE_IP="10.44.65.38"
> >>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>> Safari/537.36"]
> >>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
> >>>> INFO
> >>>>>> USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
> >>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> >>>> APP="Ignio
> >>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> >>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
> >>>>>> Joe.User@mycompantions.com RelayDomain=
> >>>> apigniodashboard-uat.mycompany.com
> >>>>>> UserName=Joe.User@mycompantions.com
> >>>>>>
> >>>>>>
> >>>>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
> >> OS X
> >>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
> >>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
> >> Mac
> >>>> OS X
> >>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>> Safari/537.36"] allow access to
> >>>> resource#0122021-03-23T16:47:20.711-05:00
> >>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >> SESSION
> >>>> INFO
> >>>>>> USER_SESSION
> >> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>> SUBJECT="Joe.User@mycompan'
> >>>>>>
> >>>>>>
> >>>>>> By inspecting the rawmsg, I can see that field four
> >> (space-delimited)
> >>>>>> indicates this is the ACCESS log. So if I were able to extract the
> >> log
> >>>>>> identifier from the msg, I could then write all access logs to the
> >> same
> >>>>>> daily file. There are other formats as well from the same device
> >> but the
> >>>>>> idea is the same.
> >>>>>>
> >>>>>> *Scott Slattery*
> >>>>>>
> >>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>
> >>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>
> >>>>>> motorolasolutions.com
> >>>>>>
> >>>>>> *O: 602.529.8226*
> >>>>>>
> >>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm> wrote:
> >>>>>>
> >>>>>>> the source logfile name is not included in the payload by the
> >> syslog
> >>>>>> spec.
> >>>>>>> It
> >>>>>>> may be in the case of your appliance, but we would need to see a
> >> sample
> >>>>>>> log to
> >>>>>>> understand ho to parse it.
> >>>>>>>
> >>>>>>> based on your template, you are using app-name, which may be listed
> >>>>>>> separtely if
> >>>>>>> it's a RFC5424 format log, or may be part of the syslog tag if
> >> it's a
> >>>>>>> RFC3164
> >>>>>>> format log over the wire (neither format has a way to specify a
> >> source
> >>>>>> log
> >>>>>>> file
> >>>>>>> by default)
> >>>>>>>
> >>>>>>> you can look at
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
> >>>>>>> and see the *-cc
> >>>>>>> options that you could apply to the app-name to eliminate control
> >>>>>>> characters.
> >>>>>>>
> >>>>>>> Again, we really need to see the original log message to understand
> >>>>>> what's
> >>>>>>> what.
> >>>>>>> Please log it with the templateRSYSLOG_DebugFormat so we can see
> >>>> exactly
> >>>>>>> what is
> >>>>>>> sent over the wire and how rsyslog has parsed it.
> >>>>>>>
> >>>>>>> David Lang
> >>>>>>>
> >>>>>>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
> >>>>>>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
> >>>>>>>> To: John Chivian <jchivian@chivian.com>
> >>>>>>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
> >>>>>>>> rsyslog-users <rsyslog@lists.adiscon.com>
> >>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>>>>>>
> >>>>>>>> Thanks, John, let me try to clarify what I mean.
> >>>>>>>>
> >>>>>>>> Normally when I forward from a remote server to the central log
> >>>>>> server, I
> >>>>>>>> can include a tag that can then be used to determine the file
> >> name I
> >>>>>> want
> >>>>>>>> on the central server. Since I have no real way to include this
> >> tag
> >>>>>> from
> >>>>>>>> the appliance, this is not an option.
> >>>>>>>>
> >>>>>>>> I'm looking for a way of inspecting the incoming packets to
> >>>> determining
> >>>>>>> the
> >>>>>>>> source logfile name (which is included in the payload) and use
> >> that
> >>>>>>>> filename on the target central server. Since there are multiple
> >> logs
> >>>>>>> being
> >>>>>>>> sent (access, audit, monitor, etc.), I'd like to segregate these
> >> into
> >>>>>>> their
> >>>>>>>> own files. I'm already using a template with the host information
> >> to
> >>>>>>>> dynamically create the file names. I just don't know how I can go
> >>>>>> beyond
> >>>>>>>> this to also include the source logname.
> >>>>>>>>
> >>>>>>>> Here's the template I'm using. It works for all other hosts where
> >> I
> >>>> can
> >>>>>>>> configure the tag but I get garbage names from the appliance. I
> >> had
> >>>>>> hoped
> >>>>>>>> that the appliance included some standard syslog tags but it
> >> doesn't
> >>>>>> seem
> >>>>>>>> so.
> >>>>>>>>
> >>>>>>>> template(name="DynRemoteLogFile" type="string"
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
> >>>>>>>>
> >>>>>>>> *Scott Slattery*
> >>>>>>>>
> >>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>
> >>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>
> >>>>>>>> motorolasolutions.com
> >>>>>>>>
> >>>>>>>> *O: 602.529.8226*
> >>>>>>>>
> >>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <
> >> jchivian@chivian.com>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Your use of the term “file name” is confusing. When senders
> >> deliver
> >>>>>> to
> >>>>>>>>> rsyslog over the network there is no exchange of files or
> >> filenames,
> >>>>>>> only
> >>>>>>>>> packets of information. Those packets are expected to be in a
> >> format
> >>>>>>> that
> >>>>>>>>> syslog understands such that useful information (header elements
> >> and
> >>>>>>>>> message body) may be parsed from them. If you as the rsyslog
> >> admin
> >>>>>>> choose
> >>>>>>>>> to use some of that header information to compose filenames for
> >>>> output
> >>>>>>>>> files, then yes you are sort of at the mercy of the senders
> >> content
> >>>>>>>>> (especially if the sender doesn’t follow the syslog rules).
> >> However,
> >>>>>>> there
> >>>>>>>>> are functions in the advanced syntax that can be used to perform
> >> the
> >>>>>>> type
> >>>>>>>>> of character replacements you’re talking about.
> >>>>>>>>>
> >>>>>>>>> It is common practice to use the syslog header/rsyslog property
> >>>>>> element
> >>>>>>>>> called “hostname” for just such purposes. Is this what you’re
> >>>> talking
> >>>>>>>>> about? You’d have to provide your configuration for real
> >> analysis,
> >>>> at
> >>>>>>>>> least the part you perceive to be responsible for the problem.
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
> >>>>>>>>> rsyslog@lists.adiscon.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>> I have a configured central log collector using rsyslog. A few
> >> of
> >>>>>> the
> >>>>>>>>>> devices forwarding their logs are appliances that have no
> >>>>>>> configuration
> >>>>>>>>>> options other than the IP forwarding address and protocol. I
> >> cannot
> >>>>>>>>> control
> >>>>>>>>>> what file names are being sent.
> >>>>>>>>>>
> >>>>>>>>>> Unfortunately, they are sending unintelligible file names with
> >>>>>>> characters
> >>>>>>>>>> that normally would be escaped. Is there any way I can control
> >> or
> >>>>>>> alter
> >>>>>>>>> the
> >>>>>>>>>> incoming file name to normalize it to avoid these odd
> >> characters?
> >>>>>>>>>>
> >>>>>>>>>> For example, could I establish a character map that maps the
> >>>>>> unallowed
> >>>>>>>>>> character to something acceptable?
> >>>>>>>>>>
> >>>>>>>>>> thanks,
> >>>>>>>>>>
> >>>>>>>>>> *Scott Slattery*
> >>>>>>>>>>
> >>>>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>>>
> >>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>>>
> >>>>>>>>>> motorolasolutions.com
> >>>>>>>>>>
> >>>>>>>>>> *O: 602.529.822*
> >>>>>>>>>>
> >>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> *For more information on how and why we collect your personal
> >>>>>>>>>> information, please visit our Privacy Policy
> >>>>>>>>>> <
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>>>>> .*
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>> rsyslog mailing list
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
> >>>>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
> >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
> >> a
> >>>>>>> myriad
> >>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> >> if
> >>>> you
> >>>>>>>>> DON'T LIKE THAT.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> *For more information on how and why we collect your personal
> >>>>>>>> information, please visit our Privacy Policy
> >>>>>>>> <
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>>> .*
> >>>>>>>> _______________________________________________
> >>>>>>>> rsyslog mailing list
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
> >>>>>>>> What's up with rsyslog? Follow
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
> >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>>>>> myriad
> >>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> >> you
> >>>>>>> DON'T LIKE THAT.
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>>
> >>>>>> *For more information on how and why we collect your personal
> >>>>>> information, please visit our Privacy Policy
> >>>>>> <
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>> .*
> >>>>>> _______________________________________________
> >>>>>> rsyslog mailing list
> >>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
> >>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
> >>>>>> What's up with rsyslog? Follow
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
> >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >> myriad
> >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> >> you
> >>>>>> DON'T LIKE THAT.
> >>>>>
> >>>>>
> >>>
> >>> --
> >>>
> >>>
> >>> *For more information on how and why we collect your personal
> >>> information, please visit our Privacy Policy
> >>> <
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>> .*
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=7hc9b2gSfvsfQzXvsRVXqGeP4Rz_tJtrykpjqkGBIdk&e=
> >>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=bDybhvts9KqI-WEfMHo7O40ulQK5M6Tg4tnKNRRIhE8&e=
> >>> What's up with rsyslog? Follow
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=zPhpcUPlK_CWLiU75_WwjyQGz5mFTiXc1yyBIfppkrI&e=
> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> DON'T LIKE THAT.
> >>
> >
> >

--


*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
you may want to capture with -X so that it decodes it into hex and you can see
newlines vs linefeeds

David Lang

On Thu, 25 Mar 2021, Scott Slattery wrote:

> Date: Thu, 25 Mar 2021 12:47:43 -0700
> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> To: David Lang <david@lang.hm>
> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
> rsyslog-users <rsyslog@lists.adiscon.com>, mariusz.kruk@safecomp.com
> Subject: Re: [rsyslog] Altering forwarded logfile names
>
> Thanks David, I just grabbed my first capture and am looking at it. I'm
> using TCP vs UDP but your comment makes sense. I'll certainly share with
> you my observations once I have some data to refer to. This has been a
> learning experience. It's also the first time I've ever seen such an issue
> with rsyslog so it's a bit of a puzzle.
>
> *Scott Slattery*
>
> *Sr. Systems & Cloud Architect*
>
> *Cloud, Compute, Information & Architecture Team*
>
> motorolasolutions.com
>
> *O: 602.529.8226*
>
> *E*: Scott.Slattery@MotorolaSolutions.com
>
>
>
>
> On Thu, Mar 25, 2021 at 12:32 PM David Lang <david@lang.hm> wrote:
>
>> the rawmsg field in the debugformat output shows exactly what rsyslog is
>> seeing.
>>
>> the reason I asked you to check multiple entries is that if rsyslog does
>> not see
>> the separator (due to either multiple messages in one UDP packet, or
>> missing
>> newlines in a TCP stream) it will combine what are intended as multiple
>> messages
>> together into one message as far as rsyslog is concerned, and only break
>> it into
>> multiple messages when it hits the configured maxmessagesize size. That
>> breaks
>> things at arbitrary points in the message, so as you look at multiple
>> messages,
>> they will not all start the same way.
>>
>> a tcpdump -s0 -A -n (with the appropriate port/ip filter) will also show
>> exactly
>> what's happening over the wire.
>>
>> David Lang
>>
>> On Thu, 25 Mar 2021, Scott Slattery wrote:
>>
>>> Date: Thu, 25 Mar 2021 12:23:03 -0700
>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>> To: Rainer Gerhards <rgerhards@hq.adiscon.com>
>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <david@lang.hm
>>> ,
>>> mariusz.kruk@safecomp.com
>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>
>>> Hey Guys, before I consume more of your time, I'm capturing the packets
>>> from the source device to determine if any of the payload is lost on its
>>> way to the rsyslog collector. After I analyze further, I'll share my
>>> observations. I'm hoping nothing is getting dropped over the wire. Thanks
>>> for your patience and help so far. I expect to have additional details
>>> shortly.
>>>
>>> *Scott Slattery*
>>>
>>> *Sr. Systems & Cloud Architect*
>>>
>>> *Cloud, Compute, Information & Architecture Team*
>>>
>>> motorolasolutions.com
>>>
>>> *O: 602.529.8226*
>>>
>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>
>>>
>>>
>>>
>>> On Thu, Mar 25, 2021 at 12:51 AM Rainer Gerhards <
>> rgerhards@hq.adiscon.com>
>>> wrote:
>>>
>>>> Sorry if I didn't catch it, but did you post the configuration? If
>>>> not, it would probably be useful.
>>>>
>>>> Looking at the message properties, this looks like the message was
>>>> received via UDP but had a LF inside it which was intended as a frame
>>>> separator. If so, that is another bug in the vendor's implementation.
>>>> Framing is used for TCP, but not UDP. For UDP it is strictly one
>>>> message per package.
>>>>
>>>> HTH,
>>>> Rainer
>>>>
>>>> El mié, 24 mar 2021 a las 22:23, Scott Slattery via rsyslog
>>>> (<rsyslog@lists.adiscon.com>) escribió:
>>>>>
>>>>> From what I can discern since there are lots of transactions, there
>> seem
>>>> to
>>>>> be multiple payload formats. For all that I have so far analyzed, the
>>>> date
>>>>> truncation issue is the same regardless of the payload received. Like
>>>> you,
>>>>> I've never used mmnormalize but I will look at it as an option. I've
>>>>> reached out to the product support team to better understand if there
>> is
>>>> a
>>>>> appliance-side misconfiguration and something that can be corrected
>> there
>>>>> or if I need to make adjustments on the rsyslog side to clean up the
>>>>> records.
>>>>>
>>>>> What I meant is that there are multiple message formats (payload data)
>>>> but
>>>>> there seems to be a consistent truncation of the date field just as you
>>>> saw
>>>>> in the app-name tag field.
>>>>>
>>>>> I appreciate your ongoing feedback and help. This issue is new to me
>> and
>>>>> will need a creative solution if the vendor has no answers.
>>>>>
>>>>> Many thanks,
>>>>>
>>>>>
>>>>> *Scott Slattery*
>>>>>
>>>>> *Sr. Systems & Cloud Architect*
>>>>>
>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>
>>>>> motorolasolutions.com
>>>>>
>>>>> *O: 602.529.8226*
>>>>>
>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Mar 24, 2021 at 2:14 PM David Lang <david@lang.hm> wrote:
>>>>>
>>>>>> when you say 'each is incorrect but the same format'
>>>>>>
>>>>>> does that mean that every log has the year missing? or that every log
>>>> is
>>>>>> combining the logs together?
>>>>>>
>>>>>> I'll note that it's possible to define a custom parser using the
>>>>>> mmnormalize
>>>>>> library and add it to the parsing stack. I helped define the need for
>>>> such
>>>>>> a
>>>>>> capability, but haven't used it yet.
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>> On Wed, 24 Mar 2021, Scott Slattery wrote:
>>>>>>
>>>>>>> Date: Wed, 24 Mar 2021 08:58:07 -0700
>>>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>>>>>> To: mariusz.kruk@safecomp.com
>>>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
>>>> david@lang.hm
>>>>>>>
>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>>>>>
>>>>>>> Yes, all the logs are coming in this way. There are three separate
>>>> logs
>>>>>>> being sent and each is incorrect in its own right but the same format
>>>>>>> behavior is consistent across each of the three logs. I will look
>>>> further
>>>>>>> into the suggestions you have provided and have already reached out
>>>> to
>>>>>> the
>>>>>>> appliance support team and will raise the nonconformance with the
>>>> RFC.
>>>>>>>
>>>>>>> This does provide me some valuable guidance and I thank each of you
>>>> for
>>>>>>> your feedback.
>>>>>>>
>>>>>>> *Scott Slattery*
>>>>>>>
>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>
>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>
>>>>>>> motorolasolutions.com
>>>>>>>
>>>>>>> *O: 602.529.8226*
>>>>>>>
>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com> wrote:
>>>>>>>
>>>>>>>> There's something strange with this log entry. Notice that it looks
>>>> as
>>>>>> if
>>>>>>>> it were more than one log entries joined with LF character (escaped
>>>> on
>>>>>>>> input as #012). And all "further" entries start with a - still
>>>>>>>> nonconformant to the RFC but "fuller" - timestamp containing
>>>> year.It's
>>>>>> the
>>>>>>>> leading event that seems to be trimmed in the front somehow. Do all
>>>>>> events
>>>>>>>> arrive like this?Wys?ano z telefonu Samsung<div>
>>>>>>>> </div><div>
>>>>>>>> </div><!-- originalMessage --><div>-------- Original message
>>>>>>>> --------</div><div>From: Scott Slattery via rsyslog <
>>>>>>>> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021 01:52
>>>>>>>> (GMT+01:00) </div><div>To: David Lang <david@lang.hm>
>>>> </div><div>Cc:
>>>>>>>> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott
>>>> Slattery
>>>>>> via
>>>>>>>> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re:
>>>> [rsyslog]
>>>>>>>> Altering forwarded logfile names </div><div>
>>>>>>>> </div>
>>>>>>>>
>>>>>>>> Hi David, fortunately I had already done this. I'm including an
>>>> actual
>>>>>> log
>>>>>>>> entry but have anonymized the data to keep the actual user and email
>>>>>>>> address confidential:
>>>>>>>>
>>>>>>>> Debug line with all properties:
>>>>>>>> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
>>>>>>>> , fromhost-ip: '10.41.102.143'
>>>>>>>> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
>>>>>>>> , PRI: 13,
>>>>>>>> syslogtag '03-23T16:'
>>>>>>>> , programname: '03-23T16'
>>>>>>>> , APP-NAME: '03-23T16'
>>>>>>>> , PROCID: '-'
>>>>>>>> , MSGID: '-'
>>>>>>>> , TIMNESTAMP: 'Mar 23 21:47:13'
>>>>>>>> , STRUCTURED-DATA: '-'
>>>>>>>> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
>>>>>> ACCESS_GATEWAY
>>>>>>>> ACCESS AUTHZ SESSION INFO USER_SESSION
>>>>>>>> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>> apigniodashboard-uat.mycompany.com"
>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>>>>> REMOTE_IP="10.44.65.38"
>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>> Safari/537.36"]
>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
>>>>>> INFO
>>>>>>>> USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
>>>>>> APP="Ignio
>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>>>>>> apigniodashboard-uat.mycompany.com
>>>>>>>> UserName=Joe.User@mycompantions.com
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
>>>> OS X
>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>>>> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
>>>> Mac
>>>>>> OS X
>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>>>> Safari/537.36"] allow access to
>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>>> SESSION
>>>>>> INFO
>>>>>>>> USER_SESSION
>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>> SUBJECT="Joe.User@mycompan'
>>>>>>>> *escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
>>>>>>>> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
>>>>>>>> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>> apigniodashboard-uat.mycompany.com"
>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>>>>> REMOTE_IP="10.44.65.38"
>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>> Safari/537.36"]
>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
>>>>>> INFO
>>>>>>>> USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
>>>>>> APP="Ignio
>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>>>>>> apigniodashboard-uat.mycompany.com
>>>>>>>> UserName=Joe.User@mycompantions.com
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
>>>> OS X
>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>>>> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
>>>> Mac
>>>>>> OS X
>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>>>> Safari/537.36"] allow access to
>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>>> SESSION
>>>>>> INFO
>>>>>>>> USER_SESSION
>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>> SUBJECT="Joe.User@mycompan'
>>>>>>>> *rawmsg*: '03-23T16:47:20.708-05:00
>>>> AUSE1OAGATST02.aws.mycompany.com
>>>>>>>> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
>>>>>>>> [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>> apigniodashboard-uat.mycompany.com"
>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>>>>> REMOTE_IP="10.44.65.38"
>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>> Safari/537.36"]
>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
>>>>>> INFO
>>>>>>>> USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>> SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
>>>>>> APP="Ignio
>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>>>>>> apigniodashboard-uat.mycompany.com
>>>>>>>> UserName=Joe.User@mycompantions.com
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
>>>> OS X
>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>>>> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
>>>> Mac
>>>>>> OS X
>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>>>> Safari/537.36"] allow access to
>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>>> SESSION
>>>>>> INFO
>>>>>>>> USER_SESSION
>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>> SUBJECT="Joe.User@mycompan'
>>>>>>>>
>>>>>>>>
>>>>>>>> By inspecting the rawmsg, I can see that field four
>>>> (space-delimited)
>>>>>>>> indicates this is the ACCESS log. So if I were able to extract the
>>>> log
>>>>>>>> identifier from the msg, I could then write all access logs to the
>>>> same
>>>>>>>> daily file. There are other formats as well from the same device
>>>> but the
>>>>>>>> idea is the same.
>>>>>>>>
>>>>>>>> *Scott Slattery*
>>>>>>>>
>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>
>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>
>>>>>>>> motorolasolutions.com
>>>>>>>>
>>>>>>>> *O: 602.529.8226*
>>>>>>>>
>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm> wrote:
>>>>>>>>
>>>>>>>>> the source logfile name is not included in the payload by the
>>>> syslog
>>>>>>>> spec.
>>>>>>>>> It
>>>>>>>>> may be in the case of your appliance, but we would need to see a
>>>> sample
>>>>>>>>> log to
>>>>>>>>> understand ho to parse it.
>>>>>>>>>
>>>>>>>>> based on your template, you are using app-name, which may be listed
>>>>>>>>> separtely if
>>>>>>>>> it's a RFC5424 format log, or may be part of the syslog tag if
>>>> it's a
>>>>>>>>> RFC3164
>>>>>>>>> format log over the wire (neither format has a way to specify a
>>>> source
>>>>>>>> log
>>>>>>>>> file
>>>>>>>>> by default)
>>>>>>>>>
>>>>>>>>> you can look at
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
>>>>>>>>> and see the *-cc
>>>>>>>>> options that you could apply to the app-name to eliminate control
>>>>>>>>> characters.
>>>>>>>>>
>>>>>>>>> Again, we really need to see the original log message to understand
>>>>>>>> what's
>>>>>>>>> what.
>>>>>>>>> Please log it with the templateRSYSLOG_DebugFormat so we can see
>>>>>> exactly
>>>>>>>>> what is
>>>>>>>>> sent over the wire and how rsyslog has parsed it.
>>>>>>>>>
>>>>>>>>> David Lang
>>>>>>>>>
>>>>>>>>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
>>>>>>>>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
>>>>>>>>>> To: John Chivian <jchivian@chivian.com>
>>>>>>>>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
>>>>>>>>>> rsyslog-users <rsyslog@lists.adiscon.com>
>>>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>>>>>>>>
>>>>>>>>>> Thanks, John, let me try to clarify what I mean.
>>>>>>>>>>
>>>>>>>>>> Normally when I forward from a remote server to the central log
>>>>>>>> server, I
>>>>>>>>>> can include a tag that can then be used to determine the file
>>>> name I
>>>>>>>> want
>>>>>>>>>> on the central server. Since I have no real way to include this
>>>> tag
>>>>>>>> from
>>>>>>>>>> the appliance, this is not an option.
>>>>>>>>>>
>>>>>>>>>> I'm looking for a way of inspecting the incoming packets to
>>>>>> determining
>>>>>>>>> the
>>>>>>>>>> source logfile name (which is included in the payload) and use
>>>> that
>>>>>>>>>> filename on the target central server. Since there are multiple
>>>> logs
>>>>>>>>> being
>>>>>>>>>> sent (access, audit, monitor, etc.), I'd like to segregate these
>>>> into
>>>>>>>>> their
>>>>>>>>>> own files. I'm already using a template with the host information
>>>> to
>>>>>>>>>> dynamically create the file names. I just don't know how I can go
>>>>>>>> beyond
>>>>>>>>>> this to also include the source logname.
>>>>>>>>>>
>>>>>>>>>> Here's the template I'm using. It works for all other hosts where
>>>> I
>>>>>> can
>>>>>>>>>> configure the tag but I get garbage names from the appliance. I
>>>> had
>>>>>>>> hoped
>>>>>>>>>> that the appliance included some standard syslog tags but it
>>>> doesn't
>>>>>>>> seem
>>>>>>>>>> so.
>>>>>>>>>>
>>>>>>>>>> template(name="DynRemoteLogFile" type="string"
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
>>>>>>>>>>
>>>>>>>>>> *Scott Slattery*
>>>>>>>>>>
>>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>>>
>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>>>
>>>>>>>>>> motorolasolutions.com
>>>>>>>>>>
>>>>>>>>>> *O: 602.529.8226*
>>>>>>>>>>
>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <
>>>> jchivian@chivian.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Your use of the term “file name” is confusing. When senders
>>>> deliver
>>>>>>>> to
>>>>>>>>>>> rsyslog over the network there is no exchange of files or
>>>> filenames,
>>>>>>>>> only
>>>>>>>>>>> packets of information. Those packets are expected to be in a
>>>> format
>>>>>>>>> that
>>>>>>>>>>> syslog understands such that useful information (header elements
>>>> and
>>>>>>>>>>> message body) may be parsed from them. If you as the rsyslog
>>>> admin
>>>>>>>>> choose
>>>>>>>>>>> to use some of that header information to compose filenames for
>>>>>> output
>>>>>>>>>>> files, then yes you are sort of at the mercy of the senders
>>>> content
>>>>>>>>>>> (especially if the sender doesn’t follow the syslog rules).
>>>> However,
>>>>>>>>> there
>>>>>>>>>>> are functions in the advanced syntax that can be used to perform
>>>> the
>>>>>>>>> type
>>>>>>>>>>> of character replacements you’re talking about.
>>>>>>>>>>>
>>>>>>>>>>> It is common practice to use the syslog header/rsyslog property
>>>>>>>> element
>>>>>>>>>>> called “hostname” for just such purposes. Is this what you’re
>>>>>> talking
>>>>>>>>>>> about? You’d have to provide your configuration for real
>>>> analysis,
>>>>>> at
>>>>>>>>>>> least the part you perceive to be responsible for the problem.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
>>>>>>>>>>> rsyslog@lists.adiscon.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I have a configured central log collector using rsyslog. A few
>>>> of
>>>>>>>> the
>>>>>>>>>>>> devices forwarding their logs are appliances that have no
>>>>>>>>> configuration
>>>>>>>>>>>> options other than the IP forwarding address and protocol. I
>>>> cannot
>>>>>>>>>>> control
>>>>>>>>>>>> what file names are being sent.
>>>>>>>>>>>>
>>>>>>>>>>>> Unfortunately, they are sending unintelligible file names with
>>>>>>>>> characters
>>>>>>>>>>>> that normally would be escaped. Is there any way I can control
>>>> or
>>>>>>>>> alter
>>>>>>>>>>> the
>>>>>>>>>>>> incoming file name to normalize it to avoid these odd
>>>> characters?
>>>>>>>>>>>>
>>>>>>>>>>>> For example, could I establish a character map that maps the
>>>>>>>> unallowed
>>>>>>>>>>>> character to something acceptable?
>>>>>>>>>>>>
>>>>>>>>>>>> thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> *Scott Slattery*
>>>>>>>>>>>>
>>>>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>>>>>
>>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>>>>>
>>>>>>>>>>>> motorolasolutions.com
>>>>>>>>>>>>
>>>>>>>>>>>> *O: 602.529.822*
>>>>>>>>>>>>
>>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *For more information on how and why we collect your personal
>>>>>>>>>>>> information, please visit our Privacy Policy
>>>>>>>>>>>> <
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>>>>>>>> .*
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
>>>>>>>>>>>> What's up with rsyslog? Follow
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>>>> a
>>>>>>>>> myriad
>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>>> if
>>>>>> you
>>>>>>>>>>> DON'T LIKE THAT.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *For more information on how and why we collect your personal
>>>>>>>>>> information, please visit our Privacy Policy
>>>>>>>>>> <
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>>>>>> .*
>>>>>>>>>> _______________________________________________
>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
>>>>>>>>>> What's up with rsyslog? Follow
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>>>> myriad
>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>>>> you
>>>>>>>>> DON'T LIKE THAT.
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>>
>>>>>>>> *For more information on how and why we collect your personal
>>>>>>>> information, please visit our Privacy Policy
>>>>>>>> <
>>>>>>>>
>>>>>>
>>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>>>>> .*
>>>>>>>> _______________________________________________
>>>>>>>> rsyslog mailing list
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
>>>>>>>> What's up with rsyslog? Follow
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>> myriad
>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>>>> you
>>>>>>>> DON'T LIKE THAT.
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>> *For more information on how and why we collect your personal
>>>>> information, please visit our Privacy Policy
>>>>> <
>>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>> .*
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=7hc9b2gSfvsfQzXvsRVXqGeP4Rz_tJtrykpjqkGBIdk&e=
>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=bDybhvts9KqI-WEfMHo7O40ulQK5M6Tg4tnKNRRIhE8&e=
>>>>> What's up with rsyslog? Follow
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=zPhpcUPlK_CWLiU75_WwjyQGz5mFTiXc1yyBIfppkrI&e=
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>>
>>>
>>>
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
Good point David, I hadn't used the -X option but your point is valid so
I'll restart my capture. Thanks again for your ongoing recommendations. ;)

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: Scott.Slattery@MotorolaSolutions.com




On Thu, Mar 25, 2021 at 1:15 PM David Lang <david@lang.hm> wrote:

> you may want to capture with -X so that it decodes it into hex and you can
> see
> newlines vs linefeeds
>
> David Lang
>
> On Thu, 25 Mar 2021, Scott Slattery wrote:
>
> > Date: Thu, 25 Mar 2021 12:47:43 -0700
> > From: Scott Slattery <scott.slattery@motorolasolutions.com>
> > To: David Lang <david@lang.hm>
> > Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
> > rsyslog-users <rsyslog@lists.adiscon.com>, mariusz.kruk@safecomp.com
> > Subject: Re: [rsyslog] Altering forwarded logfile names
> >
> > Thanks David, I just grabbed my first capture and am looking at it. I'm
> > using TCP vs UDP but your comment makes sense. I'll certainly share with
> > you my observations once I have some data to refer to. This has been a
> > learning experience. It's also the first time I've ever seen such an
> issue
> > with rsyslog so it's a bit of a puzzle.
> >
> > *Scott Slattery*
> >
> > *Sr. Systems & Cloud Architect*
> >
> > *Cloud, Compute, Information & Architecture Team*
> >
> > motorolasolutions.com
> >
> > *O: 602.529.8226*
> >
> > *E*: Scott.Slattery@MotorolaSolutions.com
> >
> >
> >
> >
> > On Thu, Mar 25, 2021 at 12:32 PM David Lang <david@lang.hm> wrote:
> >
> >> the rawmsg field in the debugformat output shows exactly what rsyslog is
> >> seeing.
> >>
> >> the reason I asked you to check multiple entries is that if rsyslog does
> >> not see
> >> the separator (due to either multiple messages in one UDP packet, or
> >> missing
> >> newlines in a TCP stream) it will combine what are intended as multiple
> >> messages
> >> together into one message as far as rsyslog is concerned, and only break
> >> it into
> >> multiple messages when it hits the configured maxmessagesize size. That
> >> breaks
> >> things at arbitrary points in the message, so as you look at multiple
> >> messages,
> >> they will not all start the same way.
> >>
> >> a tcpdump -s0 -A -n (with the appropriate port/ip filter) will also show
> >> exactly
> >> what's happening over the wire.
> >>
> >> David Lang
> >>
> >> On Thu, 25 Mar 2021, Scott Slattery wrote:
> >>
> >>> Date: Thu, 25 Mar 2021 12:23:03 -0700
> >>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> >>> To: Rainer Gerhards <rgerhards@hq.adiscon.com>
> >>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
> david@lang.hm
> >>> ,
> >>> mariusz.kruk@safecomp.com
> >>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>
> >>> Hey Guys, before I consume more of your time, I'm capturing the packets
> >>> from the source device to determine if any of the payload is lost on
> its
> >>> way to the rsyslog collector. After I analyze further, I'll share my
> >>> observations. I'm hoping nothing is getting dropped over the wire.
> Thanks
> >>> for your patience and help so far. I expect to have additional details
> >>> shortly.
> >>>
> >>> *Scott Slattery*
> >>>
> >>> *Sr. Systems & Cloud Architect*
> >>>
> >>> *Cloud, Compute, Information & Architecture Team*
> >>>
> >>> motorolasolutions.com
> >>>
> >>> *O: 602.529.8226*
> >>>
> >>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Mar 25, 2021 at 12:51 AM Rainer Gerhards <
> >> rgerhards@hq.adiscon.com>
> >>> wrote:
> >>>
> >>>> Sorry if I didn't catch it, but did you post the configuration? If
> >>>> not, it would probably be useful.
> >>>>
> >>>> Looking at the message properties, this looks like the message was
> >>>> received via UDP but had a LF inside it which was intended as a frame
> >>>> separator. If so, that is another bug in the vendor's implementation.
> >>>> Framing is used for TCP, but not UDP. For UDP it is strictly one
> >>>> message per package.
> >>>>
> >>>> HTH,
> >>>> Rainer
> >>>>
> >>>> El mié, 24 mar 2021 a las 22:23, Scott Slattery via rsyslog
> >>>> (<rsyslog@lists.adiscon.com>) escribió:
> >>>>>
> >>>>> From what I can discern since there are lots of transactions, there
> >> seem
> >>>> to
> >>>>> be multiple payload formats. For all that I have so far analyzed, the
> >>>> date
> >>>>> truncation issue is the same regardless of the payload received. Like
> >>>> you,
> >>>>> I've never used mmnormalize but I will look at it as an option. I've
> >>>>> reached out to the product support team to better understand if there
> >> is
> >>>> a
> >>>>> appliance-side misconfiguration and something that can be corrected
> >> there
> >>>>> or if I need to make adjustments on the rsyslog side to clean up the
> >>>>> records.
> >>>>>
> >>>>> What I meant is that there are multiple message formats (payload
> data)
> >>>> but
> >>>>> there seems to be a consistent truncation of the date field just as
> you
> >>>> saw
> >>>>> in the app-name tag field.
> >>>>>
> >>>>> I appreciate your ongoing feedback and help. This issue is new to me
> >> and
> >>>>> will need a creative solution if the vendor has no answers.
> >>>>>
> >>>>> Many thanks,
> >>>>>
> >>>>>
> >>>>> *Scott Slattery*
> >>>>>
> >>>>> *Sr. Systems & Cloud Architect*
> >>>>>
> >>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>
> >>>>> motorolasolutions.com
> >>>>>
> >>>>> *O: 602.529.8226*
> >>>>>
> >>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Mar 24, 2021 at 2:14 PM David Lang <david@lang.hm> wrote:
> >>>>>
> >>>>>> when you say 'each is incorrect but the same format'
> >>>>>>
> >>>>>> does that mean that every log has the year missing? or that every
> log
> >>>> is
> >>>>>> combining the logs together?
> >>>>>>
> >>>>>> I'll note that it's possible to define a custom parser using the
> >>>>>> mmnormalize
> >>>>>> library and add it to the parsing stack. I helped define the need
> for
> >>>> such
> >>>>>> a
> >>>>>> capability, but haven't used it yet.
> >>>>>>
> >>>>>> David Lang
> >>>>>>
> >>>>>> On Wed, 24 Mar 2021, Scott Slattery wrote:
> >>>>>>
> >>>>>>> Date: Wed, 24 Mar 2021 08:58:07 -0700
> >>>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> >>>>>>> To: mariusz.kruk@safecomp.com
> >>>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
> >>>> david@lang.hm
> >>>>>>>
> >>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>>>>>
> >>>>>>> Yes, all the logs are coming in this way. There are three separate
> >>>> logs
> >>>>>>> being sent and each is incorrect in its own right but the same
> format
> >>>>>>> behavior is consistent across each of the three logs. I will look
> >>>> further
> >>>>>>> into the suggestions you have provided and have already reached out
> >>>> to
> >>>>>> the
> >>>>>>> appliance support team and will raise the nonconformance with the
> >>>> RFC.
> >>>>>>>
> >>>>>>> This does provide me some valuable guidance and I thank each of you
> >>>> for
> >>>>>>> your feedback.
> >>>>>>>
> >>>>>>> *Scott Slattery*
> >>>>>>>
> >>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>
> >>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>
> >>>>>>> motorolasolutions.com
> >>>>>>>
> >>>>>>> *O: 602.529.8226*
> >>>>>>>
> >>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com>
> wrote:
> >>>>>>>
> >>>>>>>> There's something strange with this log entry. Notice that it
> looks
> >>>> as
> >>>>>> if
> >>>>>>>> it were more than one log entries joined with LF character
> (escaped
> >>>> on
> >>>>>>>> input as #012). And all "further" entries start with a - still
> >>>>>>>> nonconformant to the RFC but "fuller" - timestamp containing
> >>>> year.It's
> >>>>>> the
> >>>>>>>> leading event that seems to be trimmed in the front somehow. Do
> all
> >>>>>> events
> >>>>>>>> arrive like this?Wys?ano z telefonu Samsung<div>
> >>>>>>>> </div><div>
> >>>>>>>> </div><!-- originalMessage --><div>-------- Original message
> >>>>>>>> --------</div><div>From: Scott Slattery via rsyslog <
> >>>>>>>> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021 01:52
> >>>>>>>> (GMT+01:00) </div><div>To: David Lang <david@lang.hm>
> >>>> </div><div>Cc:
> >>>>>>>> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott
> >>>> Slattery
> >>>>>> via
> >>>>>>>> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re:
> >>>> [rsyslog]
> >>>>>>>> Altering forwarded logfile names </div><div>
> >>>>>>>> </div>
> >>>>>>>>
> >>>>>>>> Hi David, fortunately I had already done this. I'm including an
> >>>> actual
> >>>>>> log
> >>>>>>>> entry but have anonymized the data to keep the actual user and
> email
> >>>>>>>> address confidential:
> >>>>>>>>
> >>>>>>>> Debug line with all properties:
> >>>>>>>> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
> >>>>>>>> , fromhost-ip: '10.41.102.143'
> >>>>>>>> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
> >>>>>>>> , PRI: 13,
> >>>>>>>> syslogtag '03-23T16:'
> >>>>>>>> , programname: '03-23T16'
> >>>>>>>> , APP-NAME: '03-23T16'
> >>>>>>>> , PROCID: '-'
> >>>>>>>> , MSGID: '-'
> >>>>>>>> , TIMNESTAMP: 'Mar 23 21:47:13'
> >>>>>>>> , STRUCTURED-DATA: '-'
> >>>>>>>> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> >>>>>> ACCESS_GATEWAY
> >>>>>>>> ACCESS AUTHZ SESSION INFO USER_SESSION
> >>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="
> >>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>> apigniodashboard-uat.mycompany.com"
> >>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> >>>>>> REMOTE_IP="10.44.65.38"
> >>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>> Safari/537.36"]
> >>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> POLICY
> >>>>>> INFO
> >>>>>>>> USER_AUTHZ
> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>> SUBJECT="Joe.User@mycompantions.com"
> RESOURCE="/_dash-dependencies"
> >>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> >>>>>> APP="Ignio
> >>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> >>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
> oag_username=
> >>>>>>>> Joe.User@mycompantions.com RelayDomain=
> >>>>>> apigniodashboard-uat.mycompany.com
> >>>>>>>> UserName=Joe.User@mycompantions.com
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
> >>>> OS X
> >>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> Chrome/88.0.4324.192
> >>>>>>>> Safari/537.36 creationTime=1616536016811
> maxInactiveInterval=3600000
> >>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
> >>>> Mac
> >>>>>> OS X
> >>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> Chrome/88.0.4324.192
> >>>>>>>> Safari/537.36"] allow access to
> >>>>>> resource#0122021-03-23T16:47:20.711-05:00
> >>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>> SESSION
> >>>>>> INFO
> >>>>>>>> USER_SESSION
> >>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>> SUBJECT="Joe.User@mycompan'
> >>>>>>>> *escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> >>>>>>>> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
> >>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="
> >>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>> apigniodashboard-uat.mycompany.com"
> >>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> >>>>>> REMOTE_IP="10.44.65.38"
> >>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>> Safari/537.36"]
> >>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> POLICY
> >>>>>> INFO
> >>>>>>>> USER_AUTHZ
> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>> SUBJECT="Joe.User@mycompantions.com"
> RESOURCE="/_dash-dependencies"
> >>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> >>>>>> APP="Ignio
> >>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> >>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
> oag_username=
> >>>>>>>> Joe.User@mycompantions.com RelayDomain=
> >>>>>> apigniodashboard-uat.mycompany.com
> >>>>>>>> UserName=Joe.User@mycompantions.com
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
> >>>> OS X
> >>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> Chrome/88.0.4324.192
> >>>>>>>> Safari/537.36 creationTime=1616536016811
> maxInactiveInterval=3600000
> >>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
> >>>> Mac
> >>>>>> OS X
> >>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> Chrome/88.0.4324.192
> >>>>>>>> Safari/537.36"] allow access to
> >>>>>> resource#0122021-03-23T16:47:20.711-05:00
> >>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>> SESSION
> >>>>>> INFO
> >>>>>>>> USER_SESSION
> >>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>> SUBJECT="Joe.User@mycompan'
> >>>>>>>> *rawmsg*: '03-23T16:47:20.708-05:00
> >>>> AUSE1OAGATST02.aws.mycompany.com
> >>>>>>>> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
> >>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="
> >>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>> apigniodashboard-uat.mycompany.com"
> >>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> >>>>>> REMOTE_IP="10.44.65.38"
> >>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>> Safari/537.36"]
> >>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> POLICY
> >>>>>> INFO
> >>>>>>>> USER_AUTHZ
> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>> SUBJECT="Joe.User@mycompantions.com"
> RESOURCE="/_dash-dependencies"
> >>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> >>>>>> APP="Ignio
> >>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> >>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
> oag_username=
> >>>>>>>> Joe.User@mycompantions.com RelayDomain=
> >>>>>> apigniodashboard-uat.mycompany.com
> >>>>>>>> UserName=Joe.User@mycompantions.com
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
> >>>> OS X
> >>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> Chrome/88.0.4324.192
> >>>>>>>> Safari/537.36 creationTime=1616536016811
> maxInactiveInterval=3600000
> >>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
> >>>> Mac
> >>>>>> OS X
> >>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> Chrome/88.0.4324.192
> >>>>>>>> Safari/537.36"] allow access to
> >>>>>> resource#0122021-03-23T16:47:20.711-05:00
> >>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>> SESSION
> >>>>>> INFO
> >>>>>>>> USER_SESSION
> >>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>> SUBJECT="Joe.User@mycompan'
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> By inspecting the rawmsg, I can see that field four
> >>>> (space-delimited)
> >>>>>>>> indicates this is the ACCESS log. So if I were able to extract the
> >>>> log
> >>>>>>>> identifier from the msg, I could then write all access logs to the
> >>>> same
> >>>>>>>> daily file. There are other formats as well from the same device
> >>>> but the
> >>>>>>>> idea is the same.
> >>>>>>>>
> >>>>>>>> *Scott Slattery*
> >>>>>>>>
> >>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>
> >>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>
> >>>>>>>> motorolasolutions.com
> >>>>>>>>
> >>>>>>>> *O: 602.529.8226*
> >>>>>>>>
> >>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm> wrote:
> >>>>>>>>
> >>>>>>>>> the source logfile name is not included in the payload by the
> >>>> syslog
> >>>>>>>> spec.
> >>>>>>>>> It
> >>>>>>>>> may be in the case of your appliance, but we would need to see a
> >>>> sample
> >>>>>>>>> log to
> >>>>>>>>> understand ho to parse it.
> >>>>>>>>>
> >>>>>>>>> based on your template, you are using app-name, which may be
> listed
> >>>>>>>>> separtely if
> >>>>>>>>> it's a RFC5424 format log, or may be part of the syslog tag if
> >>>> it's a
> >>>>>>>>> RFC3164
> >>>>>>>>> format log over the wire (neither format has a way to specify a
> >>>> source
> >>>>>>>> log
> >>>>>>>>> file
> >>>>>>>>> by default)
> >>>>>>>>>
> >>>>>>>>> you can look at
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
> >>>>>>>>> and see the *-cc
> >>>>>>>>> options that you could apply to the app-name to eliminate control
> >>>>>>>>> characters.
> >>>>>>>>>
> >>>>>>>>> Again, we really need to see the original log message to
> understand
> >>>>>>>> what's
> >>>>>>>>> what.
> >>>>>>>>> Please log it with the templateRSYSLOG_DebugFormat so we can see
> >>>>>> exactly
> >>>>>>>>> what is
> >>>>>>>>> sent over the wire and how rsyslog has parsed it.
> >>>>>>>>>
> >>>>>>>>> David Lang
> >>>>>>>>>
> >>>>>>>>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
> >>>>>>>>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
> >>>>>>>>>> To: John Chivian <jchivian@chivian.com>
> >>>>>>>>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
> >>>>>>>>>> rsyslog-users <rsyslog@lists.adiscon.com>
> >>>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>>>>>>>>
> >>>>>>>>>> Thanks, John, let me try to clarify what I mean.
> >>>>>>>>>>
> >>>>>>>>>> Normally when I forward from a remote server to the central log
> >>>>>>>> server, I
> >>>>>>>>>> can include a tag that can then be used to determine the file
> >>>> name I
> >>>>>>>> want
> >>>>>>>>>> on the central server. Since I have no real way to include this
> >>>> tag
> >>>>>>>> from
> >>>>>>>>>> the appliance, this is not an option.
> >>>>>>>>>>
> >>>>>>>>>> I'm looking for a way of inspecting the incoming packets to
> >>>>>> determining
> >>>>>>>>> the
> >>>>>>>>>> source logfile name (which is included in the payload) and use
> >>>> that
> >>>>>>>>>> filename on the target central server. Since there are multiple
> >>>> logs
> >>>>>>>>> being
> >>>>>>>>>> sent (access, audit, monitor, etc.), I'd like to segregate these
> >>>> into
> >>>>>>>>> their
> >>>>>>>>>> own files. I'm already using a template with the host
> information
> >>>> to
> >>>>>>>>>> dynamically create the file names. I just don't know how I can
> go
> >>>>>>>> beyond
> >>>>>>>>>> this to also include the source logname.
> >>>>>>>>>>
> >>>>>>>>>> Here's the template I'm using. It works for all other hosts
> where
> >>>> I
> >>>>>> can
> >>>>>>>>>> configure the tag but I get garbage names from the appliance. I
> >>>> had
> >>>>>>>> hoped
> >>>>>>>>>> that the appliance included some standard syslog tags but it
> >>>> doesn't
> >>>>>>>> seem
> >>>>>>>>>> so.
> >>>>>>>>>>
> >>>>>>>>>> template(name="DynRemoteLogFile" type="string"
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
> >>>>>>>>>>
> >>>>>>>>>> *Scott Slattery*
> >>>>>>>>>>
> >>>>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>>>
> >>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>>>
> >>>>>>>>>> motorolasolutions.com
> >>>>>>>>>>
> >>>>>>>>>> *O: 602.529.8226*
> >>>>>>>>>>
> >>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <
> >>>> jchivian@chivian.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Your use of the term “file name” is confusing. When senders
> >>>> deliver
> >>>>>>>> to
> >>>>>>>>>>> rsyslog over the network there is no exchange of files or
> >>>> filenames,
> >>>>>>>>> only
> >>>>>>>>>>> packets of information. Those packets are expected to be in a
> >>>> format
> >>>>>>>>> that
> >>>>>>>>>>> syslog understands such that useful information (header
> elements
> >>>> and
> >>>>>>>>>>> message body) may be parsed from them. If you as the rsyslog
> >>>> admin
> >>>>>>>>> choose
> >>>>>>>>>>> to use some of that header information to compose filenames for
> >>>>>> output
> >>>>>>>>>>> files, then yes you are sort of at the mercy of the senders
> >>>> content
> >>>>>>>>>>> (especially if the sender doesn’t follow the syslog rules).
> >>>> However,
> >>>>>>>>> there
> >>>>>>>>>>> are functions in the advanced syntax that can be used to
> perform
> >>>> the
> >>>>>>>>> type
> >>>>>>>>>>> of character replacements you’re talking about.
> >>>>>>>>>>>
> >>>>>>>>>>> It is common practice to use the syslog header/rsyslog property
> >>>>>>>> element
> >>>>>>>>>>> called “hostname” for just such purposes. Is this what you’re
> >>>>>> talking
> >>>>>>>>>>> about? You’d have to provide your configuration for real
> >>>> analysis,
> >>>>>> at
> >>>>>>>>>>> least the part you perceive to be responsible for the problem.
> >>>>>>>>>>>
> >>>>>>>>>>> Regards,
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
> >>>>>>>>>>> rsyslog@lists.adiscon.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> I have a configured central log collector using rsyslog. A few
> >>>> of
> >>>>>>>> the
> >>>>>>>>>>>> devices forwarding their logs are appliances that have no
> >>>>>>>>> configuration
> >>>>>>>>>>>> options other than the IP forwarding address and protocol. I
> >>>> cannot
> >>>>>>>>>>> control
> >>>>>>>>>>>> what file names are being sent.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Unfortunately, they are sending unintelligible file names with
> >>>>>>>>> characters
> >>>>>>>>>>>> that normally would be escaped. Is there any way I can control
> >>>> or
> >>>>>>>>> alter
> >>>>>>>>>>> the
> >>>>>>>>>>>> incoming file name to normalize it to avoid these odd
> >>>> characters?
> >>>>>>>>>>>>
> >>>>>>>>>>>> For example, could I establish a character map that maps the
> >>>>>>>> unallowed
> >>>>>>>>>>>> character to something acceptable?
> >>>>>>>>>>>>
> >>>>>>>>>>>> thanks,
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Scott Slattery*
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>>>>>
> >>>>>>>>>>>> motorolasolutions.com
> >>>>>>>>>>>>
> >>>>>>>>>>>> *O: 602.529.822*
> >>>>>>>>>>>>
> >>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> *For more information on how and why we collect your personal
> >>>>>>>>>>>> information, please visit our Privacy Policy
> >>>>>>>>>>>> <
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>>>>>>> .*
> >>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>> rsyslog mailing list
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
> >>>>>>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
> >>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> by
> >>>> a
> >>>>>>>>> myriad
> >>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> >>>> if
> >>>>>> you
> >>>>>>>>>>> DON'T LIKE THAT.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> *For more information on how and why we collect your personal
> >>>>>>>>>> information, please visit our Privacy Policy
> >>>>>>>>>> <
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>>>>> .*
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>> rsyslog mailing list
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
> >>>>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
> >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
> a
> >>>>>>>> myriad
> >>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if
> >>>> you
> >>>>>>>>> DON'T LIKE THAT.
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> *For more information on how and why we collect your personal
> >>>>>>>> information, please visit our Privacy Policy
> >>>>>>>> <
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>>>> .*
> >>>>>>>> _______________________________________________
> >>>>>>>> rsyslog mailing list
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
> >>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
> >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>>> myriad
> >>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> >>>> you
> >>>>>>>> DON'T LIKE THAT.
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>> --
> >>>>>
> >>>>>
> >>>>> *For more information on how and why we collect your personal
> >>>>> information, please visit our Privacy Policy
> >>>>> <
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>> .*
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=7hc9b2gSfvsfQzXvsRVXqGeP4Rz_tJtrykpjqkGBIdk&e=
> >>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=bDybhvts9KqI-WEfMHo7O40ulQK5M6Tg4tnKNRRIhE8&e=
> >>>>> What's up with rsyslog? Follow
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=zPhpcUPlK_CWLiU75_WwjyQGz5mFTiXc1yyBIfppkrI&e=
> >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >> myriad
> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >>>> DON'T LIKE THAT.
> >>>>
> >>>
> >>>
> >
> >

--


*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
So I'm looking at the pcap and on the first log record the header
information looks fine and the record terminated by 0x0A line feed.

When I look at the record containing this record written to the rsyslog
file, the record length is 65586 (64k) long which is what I think you were
referring to earlier. Why would rsyslog write such a long record? I
initially thought my data may be getting truncated so I updated
'$MaxMessageSize 64k'. Any suggestion how I fix this behavior.

I can confirm that each log record in the pcap is terminated by 0x0A and
looks in tact so must be something I need to do on rsyslog to fix this.

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: Scott.Slattery@MotorolaSolutions.com




On Thu, Mar 25, 2021 at 1:15 PM David Lang <david@lang.hm> wrote:

> you may want to capture with -X so that it decodes it into hex and you can
> see
> newlines vs linefeeds
>
> David Lang
>
> On Thu, 25 Mar 2021, Scott Slattery wrote:
>
> > Date: Thu, 25 Mar 2021 12:47:43 -0700
> > From: Scott Slattery <scott.slattery@motorolasolutions.com>
> > To: David Lang <david@lang.hm>
> > Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
> > rsyslog-users <rsyslog@lists.adiscon.com>, mariusz.kruk@safecomp.com
> > Subject: Re: [rsyslog] Altering forwarded logfile names
> >
> > Thanks David, I just grabbed my first capture and am looking at it. I'm
> > using TCP vs UDP but your comment makes sense. I'll certainly share with
> > you my observations once I have some data to refer to. This has been a
> > learning experience. It's also the first time I've ever seen such an
> issue
> > with rsyslog so it's a bit of a puzzle.
> >
> > *Scott Slattery*
> >
> > *Sr. Systems & Cloud Architect*
> >
> > *Cloud, Compute, Information & Architecture Team*
> >
> > motorolasolutions.com
> >
> > *O: 602.529.8226*
> >
> > *E*: Scott.Slattery@MotorolaSolutions.com
> >
> >
> >
> >
> > On Thu, Mar 25, 2021 at 12:32 PM David Lang <david@lang.hm> wrote:
> >
> >> the rawmsg field in the debugformat output shows exactly what rsyslog is
> >> seeing.
> >>
> >> the reason I asked you to check multiple entries is that if rsyslog does
> >> not see
> >> the separator (due to either multiple messages in one UDP packet, or
> >> missing
> >> newlines in a TCP stream) it will combine what are intended as multiple
> >> messages
> >> together into one message as far as rsyslog is concerned, and only break
> >> it into
> >> multiple messages when it hits the configured maxmessagesize size. That
> >> breaks
> >> things at arbitrary points in the message, so as you look at multiple
> >> messages,
> >> they will not all start the same way.
> >>
> >> a tcpdump -s0 -A -n (with the appropriate port/ip filter) will also show
> >> exactly
> >> what's happening over the wire.
> >>
> >> David Lang
> >>
> >> On Thu, 25 Mar 2021, Scott Slattery wrote:
> >>
> >>> Date: Thu, 25 Mar 2021 12:23:03 -0700
> >>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> >>> To: Rainer Gerhards <rgerhards@hq.adiscon.com>
> >>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
> david@lang.hm
> >>> ,
> >>> mariusz.kruk@safecomp.com
> >>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>
> >>> Hey Guys, before I consume more of your time, I'm capturing the packets
> >>> from the source device to determine if any of the payload is lost on
> its
> >>> way to the rsyslog collector. After I analyze further, I'll share my
> >>> observations. I'm hoping nothing is getting dropped over the wire.
> Thanks
> >>> for your patience and help so far. I expect to have additional details
> >>> shortly.
> >>>
> >>> *Scott Slattery*
> >>>
> >>> *Sr. Systems & Cloud Architect*
> >>>
> >>> *Cloud, Compute, Information & Architecture Team*
> >>>
> >>> motorolasolutions.com
> >>>
> >>> *O: 602.529.8226*
> >>>
> >>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Mar 25, 2021 at 12:51 AM Rainer Gerhards <
> >> rgerhards@hq.adiscon.com>
> >>> wrote:
> >>>
> >>>> Sorry if I didn't catch it, but did you post the configuration? If
> >>>> not, it would probably be useful.
> >>>>
> >>>> Looking at the message properties, this looks like the message was
> >>>> received via UDP but had a LF inside it which was intended as a frame
> >>>> separator. If so, that is another bug in the vendor's implementation.
> >>>> Framing is used for TCP, but not UDP. For UDP it is strictly one
> >>>> message per package.
> >>>>
> >>>> HTH,
> >>>> Rainer
> >>>>
> >>>> El mié, 24 mar 2021 a las 22:23, Scott Slattery via rsyslog
> >>>> (<rsyslog@lists.adiscon.com>) escribió:
> >>>>>
> >>>>> From what I can discern since there are lots of transactions, there
> >> seem
> >>>> to
> >>>>> be multiple payload formats. For all that I have so far analyzed, the
> >>>> date
> >>>>> truncation issue is the same regardless of the payload received. Like
> >>>> you,
> >>>>> I've never used mmnormalize but I will look at it as an option. I've
> >>>>> reached out to the product support team to better understand if there
> >> is
> >>>> a
> >>>>> appliance-side misconfiguration and something that can be corrected
> >> there
> >>>>> or if I need to make adjustments on the rsyslog side to clean up the
> >>>>> records.
> >>>>>
> >>>>> What I meant is that there are multiple message formats (payload
> data)
> >>>> but
> >>>>> there seems to be a consistent truncation of the date field just as
> you
> >>>> saw
> >>>>> in the app-name tag field.
> >>>>>
> >>>>> I appreciate your ongoing feedback and help. This issue is new to me
> >> and
> >>>>> will need a creative solution if the vendor has no answers.
> >>>>>
> >>>>> Many thanks,
> >>>>>
> >>>>>
> >>>>> *Scott Slattery*
> >>>>>
> >>>>> *Sr. Systems & Cloud Architect*
> >>>>>
> >>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>
> >>>>> motorolasolutions.com
> >>>>>
> >>>>> *O: 602.529.8226*
> >>>>>
> >>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Mar 24, 2021 at 2:14 PM David Lang <david@lang.hm> wrote:
> >>>>>
> >>>>>> when you say 'each is incorrect but the same format'
> >>>>>>
> >>>>>> does that mean that every log has the year missing? or that every
> log
> >>>> is
> >>>>>> combining the logs together?
> >>>>>>
> >>>>>> I'll note that it's possible to define a custom parser using the
> >>>>>> mmnormalize
> >>>>>> library and add it to the parsing stack. I helped define the need
> for
> >>>> such
> >>>>>> a
> >>>>>> capability, but haven't used it yet.
> >>>>>>
> >>>>>> David Lang
> >>>>>>
> >>>>>> On Wed, 24 Mar 2021, Scott Slattery wrote:
> >>>>>>
> >>>>>>> Date: Wed, 24 Mar 2021 08:58:07 -0700
> >>>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> >>>>>>> To: mariusz.kruk@safecomp.com
> >>>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
> >>>> david@lang.hm
> >>>>>>>
> >>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>>>>>
> >>>>>>> Yes, all the logs are coming in this way. There are three separate
> >>>> logs
> >>>>>>> being sent and each is incorrect in its own right but the same
> format
> >>>>>>> behavior is consistent across each of the three logs. I will look
> >>>> further
> >>>>>>> into the suggestions you have provided and have already reached out
> >>>> to
> >>>>>> the
> >>>>>>> appliance support team and will raise the nonconformance with the
> >>>> RFC.
> >>>>>>>
> >>>>>>> This does provide me some valuable guidance and I thank each of you
> >>>> for
> >>>>>>> your feedback.
> >>>>>>>
> >>>>>>> *Scott Slattery*
> >>>>>>>
> >>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>
> >>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>
> >>>>>>> motorolasolutions.com
> >>>>>>>
> >>>>>>> *O: 602.529.8226*
> >>>>>>>
> >>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com>
> wrote:
> >>>>>>>
> >>>>>>>> There's something strange with this log entry. Notice that it
> looks
> >>>> as
> >>>>>> if
> >>>>>>>> it were more than one log entries joined with LF character
> (escaped
> >>>> on
> >>>>>>>> input as #012). And all "further" entries start with a - still
> >>>>>>>> nonconformant to the RFC but "fuller" - timestamp containing
> >>>> year.It's
> >>>>>> the
> >>>>>>>> leading event that seems to be trimmed in the front somehow. Do
> all
> >>>>>> events
> >>>>>>>> arrive like this?Wys?ano z telefonu Samsung<div>
> >>>>>>>> </div><div>
> >>>>>>>> </div><!-- originalMessage --><div>-------- Original message
> >>>>>>>> --------</div><div>From: Scott Slattery via rsyslog <
> >>>>>>>> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021 01:52
> >>>>>>>> (GMT+01:00) </div><div>To: David Lang <david@lang.hm>
> >>>> </div><div>Cc:
> >>>>>>>> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott
> >>>> Slattery
> >>>>>> via
> >>>>>>>> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re:
> >>>> [rsyslog]
> >>>>>>>> Altering forwarded logfile names </div><div>
> >>>>>>>> </div>
> >>>>>>>>
> >>>>>>>> Hi David, fortunately I had already done this. I'm including an
> >>>> actual
> >>>>>> log
> >>>>>>>> entry but have anonymized the data to keep the actual user and
> email
> >>>>>>>> address confidential:
> >>>>>>>>
> >>>>>>>> Debug line with all properties:
> >>>>>>>> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
> >>>>>>>> , fromhost-ip: '10.41.102.143'
> >>>>>>>> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
> >>>>>>>> , PRI: 13,
> >>>>>>>> syslogtag '03-23T16:'
> >>>>>>>> , programname: '03-23T16'
> >>>>>>>> , APP-NAME: '03-23T16'
> >>>>>>>> , PROCID: '-'
> >>>>>>>> , MSGID: '-'
> >>>>>>>> , TIMNESTAMP: 'Mar 23 21:47:13'
> >>>>>>>> , STRUCTURED-DATA: '-'
> >>>>>>>> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> >>>>>> ACCESS_GATEWAY
> >>>>>>>> ACCESS AUTHZ SESSION INFO USER_SESSION
> >>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="
> >>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>> apigniodashboard-uat.mycompany.com"
> >>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> >>>>>> REMOTE_IP="10.44.65.38"
> >>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>> Safari/537.36"]
> >>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> POLICY
> >>>>>> INFO
> >>>>>>>> USER_AUTHZ
> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>> SUBJECT="Joe.User@mycompantions.com"
> RESOURCE="/_dash-dependencies"
> >>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> >>>>>> APP="Ignio
> >>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> >>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
> oag_username=
> >>>>>>>> Joe.User@mycompantions.com RelayDomain=
> >>>>>> apigniodashboard-uat.mycompany.com
> >>>>>>>> UserName=Joe.User@mycompantions.com
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
> >>>> OS X
> >>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> Chrome/88.0.4324.192
> >>>>>>>> Safari/537.36 creationTime=1616536016811
> maxInactiveInterval=3600000
> >>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
> >>>> Mac
> >>>>>> OS X
> >>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> Chrome/88.0.4324.192
> >>>>>>>> Safari/537.36"] allow access to
> >>>>>> resource#0122021-03-23T16:47:20.711-05:00
> >>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>> SESSION
> >>>>>> INFO
> >>>>>>>> USER_SESSION
> >>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>> SUBJECT="Joe.User@mycompan'
> >>>>>>>> *escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> >>>>>>>> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
> >>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="
> >>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>> apigniodashboard-uat.mycompany.com"
> >>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> >>>>>> REMOTE_IP="10.44.65.38"
> >>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>> Safari/537.36"]
> >>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> POLICY
> >>>>>> INFO
> >>>>>>>> USER_AUTHZ
> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>> SUBJECT="Joe.User@mycompantions.com"
> RESOURCE="/_dash-dependencies"
> >>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> >>>>>> APP="Ignio
> >>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> >>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
> oag_username=
> >>>>>>>> Joe.User@mycompantions.com RelayDomain=
> >>>>>> apigniodashboard-uat.mycompany.com
> >>>>>>>> UserName=Joe.User@mycompantions.com
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
> >>>> OS X
> >>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> Chrome/88.0.4324.192
> >>>>>>>> Safari/537.36 creationTime=1616536016811
> maxInactiveInterval=3600000
> >>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
> >>>> Mac
> >>>>>> OS X
> >>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> Chrome/88.0.4324.192
> >>>>>>>> Safari/537.36"] allow access to
> >>>>>> resource#0122021-03-23T16:47:20.711-05:00
> >>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>> SESSION
> >>>>>> INFO
> >>>>>>>> USER_SESSION
> >>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>> SUBJECT="Joe.User@mycompan'
> >>>>>>>> *rawmsg*: '03-23T16:47:20.708-05:00
> >>>> AUSE1OAGATST02.aws.mycompany.com
> >>>>>>>> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
> >>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> SUBJECT="
> >>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>> apigniodashboard-uat.mycompany.com"
> >>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> >>>>>> REMOTE_IP="10.44.65.38"
> >>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>> Safari/537.36"]
> >>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> POLICY
> >>>>>> INFO
> >>>>>>>> USER_AUTHZ
> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>> SUBJECT="Joe.User@mycompantions.com"
> RESOURCE="/_dash-dependencies"
> >>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> >>>>>> APP="Ignio
> >>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
> >>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
> oag_username=
> >>>>>>>> Joe.User@mycompantions.com RelayDomain=
> >>>>>> apigniodashboard-uat.mycompany.com
> >>>>>>>> UserName=Joe.User@mycompantions.com
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
> >>>> OS X
> >>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> Chrome/88.0.4324.192
> >>>>>>>> Safari/537.36 creationTime=1616536016811
> maxInactiveInterval=3600000
> >>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
> >>>> Mac
> >>>>>> OS X
> >>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> Chrome/88.0.4324.192
> >>>>>>>> Safari/537.36"] allow access to
> >>>>>> resource#0122021-03-23T16:47:20.711-05:00
> >>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>> SESSION
> >>>>>> INFO
> >>>>>>>> USER_SESSION
> >>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>> SUBJECT="Joe.User@mycompan'
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> By inspecting the rawmsg, I can see that field four
> >>>> (space-delimited)
> >>>>>>>> indicates this is the ACCESS log. So if I were able to extract the
> >>>> log
> >>>>>>>> identifier from the msg, I could then write all access logs to the
> >>>> same
> >>>>>>>> daily file. There are other formats as well from the same device
> >>>> but the
> >>>>>>>> idea is the same.
> >>>>>>>>
> >>>>>>>> *Scott Slattery*
> >>>>>>>>
> >>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>
> >>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>
> >>>>>>>> motorolasolutions.com
> >>>>>>>>
> >>>>>>>> *O: 602.529.8226*
> >>>>>>>>
> >>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm> wrote:
> >>>>>>>>
> >>>>>>>>> the source logfile name is not included in the payload by the
> >>>> syslog
> >>>>>>>> spec.
> >>>>>>>>> It
> >>>>>>>>> may be in the case of your appliance, but we would need to see a
> >>>> sample
> >>>>>>>>> log to
> >>>>>>>>> understand ho to parse it.
> >>>>>>>>>
> >>>>>>>>> based on your template, you are using app-name, which may be
> listed
> >>>>>>>>> separtely if
> >>>>>>>>> it's a RFC5424 format log, or may be part of the syslog tag if
> >>>> it's a
> >>>>>>>>> RFC3164
> >>>>>>>>> format log over the wire (neither format has a way to specify a
> >>>> source
> >>>>>>>> log
> >>>>>>>>> file
> >>>>>>>>> by default)
> >>>>>>>>>
> >>>>>>>>> you can look at
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
> >>>>>>>>> and see the *-cc
> >>>>>>>>> options that you could apply to the app-name to eliminate control
> >>>>>>>>> characters.
> >>>>>>>>>
> >>>>>>>>> Again, we really need to see the original log message to
> understand
> >>>>>>>> what's
> >>>>>>>>> what.
> >>>>>>>>> Please log it with the templateRSYSLOG_DebugFormat so we can see
> >>>>>> exactly
> >>>>>>>>> what is
> >>>>>>>>> sent over the wire and how rsyslog has parsed it.
> >>>>>>>>>
> >>>>>>>>> David Lang
> >>>>>>>>>
> >>>>>>>>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
> >>>>>>>>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
> >>>>>>>>>> To: John Chivian <jchivian@chivian.com>
> >>>>>>>>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
> >>>>>>>>>> rsyslog-users <rsyslog@lists.adiscon.com>
> >>>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>>>>>>>>
> >>>>>>>>>> Thanks, John, let me try to clarify what I mean.
> >>>>>>>>>>
> >>>>>>>>>> Normally when I forward from a remote server to the central log
> >>>>>>>> server, I
> >>>>>>>>>> can include a tag that can then be used to determine the file
> >>>> name I
> >>>>>>>> want
> >>>>>>>>>> on the central server. Since I have no real way to include this
> >>>> tag
> >>>>>>>> from
> >>>>>>>>>> the appliance, this is not an option.
> >>>>>>>>>>
> >>>>>>>>>> I'm looking for a way of inspecting the incoming packets to
> >>>>>> determining
> >>>>>>>>> the
> >>>>>>>>>> source logfile name (which is included in the payload) and use
> >>>> that
> >>>>>>>>>> filename on the target central server. Since there are multiple
> >>>> logs
> >>>>>>>>> being
> >>>>>>>>>> sent (access, audit, monitor, etc.), I'd like to segregate these
> >>>> into
> >>>>>>>>> their
> >>>>>>>>>> own files. I'm already using a template with the host
> information
> >>>> to
> >>>>>>>>>> dynamically create the file names. I just don't know how I can
> go
> >>>>>>>> beyond
> >>>>>>>>>> this to also include the source logname.
> >>>>>>>>>>
> >>>>>>>>>> Here's the template I'm using. It works for all other hosts
> where
> >>>> I
> >>>>>> can
> >>>>>>>>>> configure the tag but I get garbage names from the appliance. I
> >>>> had
> >>>>>>>> hoped
> >>>>>>>>>> that the appliance included some standard syslog tags but it
> >>>> doesn't
> >>>>>>>> seem
> >>>>>>>>>> so.
> >>>>>>>>>>
> >>>>>>>>>> template(name="DynRemoteLogFile" type="string"
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
> >>>>>>>>>>
> >>>>>>>>>> *Scott Slattery*
> >>>>>>>>>>
> >>>>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>>>
> >>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>>>
> >>>>>>>>>> motorolasolutions.com
> >>>>>>>>>>
> >>>>>>>>>> *O: 602.529.8226*
> >>>>>>>>>>
> >>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <
> >>>> jchivian@chivian.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Your use of the term “file name” is confusing. When senders
> >>>> deliver
> >>>>>>>> to
> >>>>>>>>>>> rsyslog over the network there is no exchange of files or
> >>>> filenames,
> >>>>>>>>> only
> >>>>>>>>>>> packets of information. Those packets are expected to be in a
> >>>> format
> >>>>>>>>> that
> >>>>>>>>>>> syslog understands such that useful information (header
> elements
> >>>> and
> >>>>>>>>>>> message body) may be parsed from them. If you as the rsyslog
> >>>> admin
> >>>>>>>>> choose
> >>>>>>>>>>> to use some of that header information to compose filenames for
> >>>>>> output
> >>>>>>>>>>> files, then yes you are sort of at the mercy of the senders
> >>>> content
> >>>>>>>>>>> (especially if the sender doesn’t follow the syslog rules).
> >>>> However,
> >>>>>>>>> there
> >>>>>>>>>>> are functions in the advanced syntax that can be used to
> perform
> >>>> the
> >>>>>>>>> type
> >>>>>>>>>>> of character replacements you’re talking about.
> >>>>>>>>>>>
> >>>>>>>>>>> It is common practice to use the syslog header/rsyslog property
> >>>>>>>> element
> >>>>>>>>>>> called “hostname” for just such purposes. Is this what you’re
> >>>>>> talking
> >>>>>>>>>>> about? You’d have to provide your configuration for real
> >>>> analysis,
> >>>>>> at
> >>>>>>>>>>> least the part you perceive to be responsible for the problem.
> >>>>>>>>>>>
> >>>>>>>>>>> Regards,
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
> >>>>>>>>>>> rsyslog@lists.adiscon.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> I have a configured central log collector using rsyslog. A few
> >>>> of
> >>>>>>>> the
> >>>>>>>>>>>> devices forwarding their logs are appliances that have no
> >>>>>>>>> configuration
> >>>>>>>>>>>> options other than the IP forwarding address and protocol. I
> >>>> cannot
> >>>>>>>>>>> control
> >>>>>>>>>>>> what file names are being sent.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Unfortunately, they are sending unintelligible file names with
> >>>>>>>>> characters
> >>>>>>>>>>>> that normally would be escaped. Is there any way I can control
> >>>> or
> >>>>>>>>> alter
> >>>>>>>>>>> the
> >>>>>>>>>>>> incoming file name to normalize it to avoid these odd
> >>>> characters?
> >>>>>>>>>>>>
> >>>>>>>>>>>> For example, could I establish a character map that maps the
> >>>>>>>> unallowed
> >>>>>>>>>>>> character to something acceptable?
> >>>>>>>>>>>>
> >>>>>>>>>>>> thanks,
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Scott Slattery*
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>>>>>
> >>>>>>>>>>>> motorolasolutions.com
> >>>>>>>>>>>>
> >>>>>>>>>>>> *O: 602.529.822*
> >>>>>>>>>>>>
> >>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> *For more information on how and why we collect your personal
> >>>>>>>>>>>> information, please visit our Privacy Policy
> >>>>>>>>>>>> <
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>>>>>>> .*
> >>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>> rsyslog mailing list
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
> >>>>>>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
> >>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> by
> >>>> a
> >>>>>>>>> myriad
> >>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> >>>> if
> >>>>>> you
> >>>>>>>>>>> DON'T LIKE THAT.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> *For more information on how and why we collect your personal
> >>>>>>>>>> information, please visit our Privacy Policy
> >>>>>>>>>> <
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>>>>> .*
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>> rsyslog mailing list
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
> >>>>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
> >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
> a
> >>>>>>>> myriad
> >>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if
> >>>> you
> >>>>>>>>> DON'T LIKE THAT.
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> *For more information on how and why we collect your personal
> >>>>>>>> information, please visit our Privacy Policy
> >>>>>>>> <
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>>>> .*
> >>>>>>>> _______________________________________________
> >>>>>>>> rsyslog mailing list
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
> >>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
> >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>>> myriad
> >>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> >>>> you
> >>>>>>>> DON'T LIKE THAT.
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>> --
> >>>>>
> >>>>>
> >>>>> *For more information on how and why we collect your personal
> >>>>> information, please visit our Privacy Policy
> >>>>> <
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>> .*
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=7hc9b2gSfvsfQzXvsRVXqGeP4Rz_tJtrykpjqkGBIdk&e=
> >>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=bDybhvts9KqI-WEfMHo7O40ulQK5M6Tg4tnKNRRIhE8&e=
> >>>>> What's up with rsyslog? Follow
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=zPhpcUPlK_CWLiU75_WwjyQGz5mFTiXc1yyBIfppkrI&e=
> >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >> myriad
> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >>>> DON'T LIKE THAT.
> >>>>
> >>>
> >>>
> >
> >

--


*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
what version of rsyslog are you running. can you post your full config?

if you are receiving via TCP and it's not splitting the logs based on newlines,
something very odd is happening.

David Lang

On Thu, 25 Mar 2021, Scott Slattery wrote:

> Date: Thu, 25 Mar 2021 15:07:22 -0700
> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> To: David Lang <david@lang.hm>
> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
> rsyslog-users <rsyslog@lists.adiscon.com>, mariusz.kruk@safecomp.com
> Subject: Re: [rsyslog] Altering forwarded logfile names
>
> So I'm looking at the pcap and on the first log record the header
> information looks fine and the record terminated by 0x0A line feed.
>
> When I look at the record containing this record written to the rsyslog
> file, the record length is 65586 (64k) long which is what I think you were
> referring to earlier. Why would rsyslog write such a long record? I
> initially thought my data may be getting truncated so I updated
> '$MaxMessageSize 64k'. Any suggestion how I fix this behavior.
>
> I can confirm that each log record in the pcap is terminated by 0x0A and
> looks in tact so must be something I need to do on rsyslog to fix this.
>
> *Scott Slattery*
>
> *Sr. Systems & Cloud Architect*
>
> *Cloud, Compute, Information & Architecture Team*
>
> motorolasolutions.com
>
> *O: 602.529.8226*
>
> *E*: Scott.Slattery@MotorolaSolutions.com
>
>
>
>
> On Thu, Mar 25, 2021 at 1:15 PM David Lang <david@lang.hm> wrote:
>
>> you may want to capture with -X so that it decodes it into hex and you can
>> see
>> newlines vs linefeeds
>>
>> David Lang
>>
>> On Thu, 25 Mar 2021, Scott Slattery wrote:
>>
>>> Date: Thu, 25 Mar 2021 12:47:43 -0700
>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>> To: David Lang <david@lang.hm>
>>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
>>> rsyslog-users <rsyslog@lists.adiscon.com>, mariusz.kruk@safecomp.com
>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>
>>> Thanks David, I just grabbed my first capture and am looking at it. I'm
>>> using TCP vs UDP but your comment makes sense. I'll certainly share with
>>> you my observations once I have some data to refer to. This has been a
>>> learning experience. It's also the first time I've ever seen such an
>> issue
>>> with rsyslog so it's a bit of a puzzle.
>>>
>>> *Scott Slattery*
>>>
>>> *Sr. Systems & Cloud Architect*
>>>
>>> *Cloud, Compute, Information & Architecture Team*
>>>
>>> motorolasolutions.com
>>>
>>> *O: 602.529.8226*
>>>
>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>
>>>
>>>
>>>
>>> On Thu, Mar 25, 2021 at 12:32 PM David Lang <david@lang.hm> wrote:
>>>
>>>> the rawmsg field in the debugformat output shows exactly what rsyslog is
>>>> seeing.
>>>>
>>>> the reason I asked you to check multiple entries is that if rsyslog does
>>>> not see
>>>> the separator (due to either multiple messages in one UDP packet, or
>>>> missing
>>>> newlines in a TCP stream) it will combine what are intended as multiple
>>>> messages
>>>> together into one message as far as rsyslog is concerned, and only break
>>>> it into
>>>> multiple messages when it hits the configured maxmessagesize size. That
>>>> breaks
>>>> things at arbitrary points in the message, so as you look at multiple
>>>> messages,
>>>> they will not all start the same way.
>>>>
>>>> a tcpdump -s0 -A -n (with the appropriate port/ip filter) will also show
>>>> exactly
>>>> what's happening over the wire.
>>>>
>>>> David Lang
>>>>
>>>> On Thu, 25 Mar 2021, Scott Slattery wrote:
>>>>
>>>>> Date: Thu, 25 Mar 2021 12:23:03 -0700
>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>>>> To: Rainer Gerhards <rgerhards@hq.adiscon.com>
>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
>> david@lang.hm
>>>>> ,
>>>>> mariusz.kruk@safecomp.com
>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>>>
>>>>> Hey Guys, before I consume more of your time, I'm capturing the packets
>>>>> from the source device to determine if any of the payload is lost on
>> its
>>>>> way to the rsyslog collector. After I analyze further, I'll share my
>>>>> observations. I'm hoping nothing is getting dropped over the wire.
>> Thanks
>>>>> for your patience and help so far. I expect to have additional details
>>>>> shortly.
>>>>>
>>>>> *Scott Slattery*
>>>>>
>>>>> *Sr. Systems & Cloud Architect*
>>>>>
>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>
>>>>> motorolasolutions.com
>>>>>
>>>>> *O: 602.529.8226*
>>>>>
>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Mar 25, 2021 at 12:51 AM Rainer Gerhards <
>>>> rgerhards@hq.adiscon.com>
>>>>> wrote:
>>>>>
>>>>>> Sorry if I didn't catch it, but did you post the configuration? If
>>>>>> not, it would probably be useful.
>>>>>>
>>>>>> Looking at the message properties, this looks like the message was
>>>>>> received via UDP but had a LF inside it which was intended as a frame
>>>>>> separator. If so, that is another bug in the vendor's implementation.
>>>>>> Framing is used for TCP, but not UDP. For UDP it is strictly one
>>>>>> message per package.
>>>>>>
>>>>>> HTH,
>>>>>> Rainer
>>>>>>
>>>>>> El mié, 24 mar 2021 a las 22:23, Scott Slattery via rsyslog
>>>>>> (<rsyslog@lists.adiscon.com>) escribió:
>>>>>>>
>>>>>>> From what I can discern since there are lots of transactions, there
>>>> seem
>>>>>> to
>>>>>>> be multiple payload formats. For all that I have so far analyzed, the
>>>>>> date
>>>>>>> truncation issue is the same regardless of the payload received. Like
>>>>>> you,
>>>>>>> I've never used mmnormalize but I will look at it as an option. I've
>>>>>>> reached out to the product support team to better understand if there
>>>> is
>>>>>> a
>>>>>>> appliance-side misconfiguration and something that can be corrected
>>>> there
>>>>>>> or if I need to make adjustments on the rsyslog side to clean up the
>>>>>>> records.
>>>>>>>
>>>>>>> What I meant is that there are multiple message formats (payload
>> data)
>>>>>> but
>>>>>>> there seems to be a consistent truncation of the date field just as
>> you
>>>>>> saw
>>>>>>> in the app-name tag field.
>>>>>>>
>>>>>>> I appreciate your ongoing feedback and help. This issue is new to me
>>>> and
>>>>>>> will need a creative solution if the vendor has no answers.
>>>>>>>
>>>>>>> Many thanks,
>>>>>>>
>>>>>>>
>>>>>>> *Scott Slattery*
>>>>>>>
>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>
>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>
>>>>>>> motorolasolutions.com
>>>>>>>
>>>>>>> *O: 602.529.8226*
>>>>>>>
>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 24, 2021 at 2:14 PM David Lang <david@lang.hm> wrote:
>>>>>>>
>>>>>>>> when you say 'each is incorrect but the same format'
>>>>>>>>
>>>>>>>> does that mean that every log has the year missing? or that every
>> log
>>>>>> is
>>>>>>>> combining the logs together?
>>>>>>>>
>>>>>>>> I'll note that it's possible to define a custom parser using the
>>>>>>>> mmnormalize
>>>>>>>> library and add it to the parsing stack. I helped define the need
>> for
>>>>>> such
>>>>>>>> a
>>>>>>>> capability, but haven't used it yet.
>>>>>>>>
>>>>>>>> David Lang
>>>>>>>>
>>>>>>>> On Wed, 24 Mar 2021, Scott Slattery wrote:
>>>>>>>>
>>>>>>>>> Date: Wed, 24 Mar 2021 08:58:07 -0700
>>>>>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>>>>>>>> To: mariusz.kruk@safecomp.com
>>>>>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
>>>>>> david@lang.hm
>>>>>>>>>
>>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>>>>>>>
>>>>>>>>> Yes, all the logs are coming in this way. There are three separate
>>>>>> logs
>>>>>>>>> being sent and each is incorrect in its own right but the same
>> format
>>>>>>>>> behavior is consistent across each of the three logs. I will look
>>>>>> further
>>>>>>>>> into the suggestions you have provided and have already reached out
>>>>>> to
>>>>>>>> the
>>>>>>>>> appliance support team and will raise the nonconformance with the
>>>>>> RFC.
>>>>>>>>>
>>>>>>>>> This does provide me some valuable guidance and I thank each of you
>>>>>> for
>>>>>>>>> your feedback.
>>>>>>>>>
>>>>>>>>> *Scott Slattery*
>>>>>>>>>
>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>>
>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>>
>>>>>>>>> motorolasolutions.com
>>>>>>>>>
>>>>>>>>> *O: 602.529.8226*
>>>>>>>>>
>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com>
>> wrote:
>>>>>>>>>
>>>>>>>>>> There's something strange with this log entry. Notice that it
>> looks
>>>>>> as
>>>>>>>> if
>>>>>>>>>> it were more than one log entries joined with LF character
>> (escaped
>>>>>> on
>>>>>>>>>> input as #012). And all "further" entries start with a - still
>>>>>>>>>> nonconformant to the RFC but "fuller" - timestamp containing
>>>>>> year.It's
>>>>>>>> the
>>>>>>>>>> leading event that seems to be trimmed in the front somehow. Do
>> all
>>>>>>>> events
>>>>>>>>>> arrive like this?Wys?ano z telefonu Samsung<div>
>>>>>>>>>> </div><div>
>>>>>>>>>> </div><!-- originalMessage --><div>-------- Original message
>>>>>>>>>> --------</div><div>From: Scott Slattery via rsyslog <
>>>>>>>>>> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021 01:52
>>>>>>>>>> (GMT+01:00) </div><div>To: David Lang <david@lang.hm>
>>>>>> </div><div>Cc:
>>>>>>>>>> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott
>>>>>> Slattery
>>>>>>>> via
>>>>>>>>>> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re:
>>>>>> [rsyslog]
>>>>>>>>>> Altering forwarded logfile names </div><div>
>>>>>>>>>> </div>
>>>>>>>>>>
>>>>>>>>>> Hi David, fortunately I had already done this. I'm including an
>>>>>> actual
>>>>>>>> log
>>>>>>>>>> entry but have anonymized the data to keep the actual user and
>> email
>>>>>>>>>> address confidential:
>>>>>>>>>>
>>>>>>>>>> Debug line with all properties:
>>>>>>>>>> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
>>>>>>>>>> , fromhost-ip: '10.41.102.143'
>>>>>>>>>> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
>>>>>>>>>> , PRI: 13,
>>>>>>>>>> syslogtag '03-23T16:'
>>>>>>>>>> , programname: '03-23T16'
>>>>>>>>>> , APP-NAME: '03-23T16'
>>>>>>>>>> , PROCID: '-'
>>>>>>>>>> , MSGID: '-'
>>>>>>>>>> , TIMNESTAMP: 'Mar 23 21:47:13'
>>>>>>>>>> , STRUCTURED-DATA: '-'
>>>>>>>>>> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
>>>>>>>> ACCESS_GATEWAY
>>>>>>>>>> ACCESS AUTHZ SESSION INFO USER_SESSION
>>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> SUBJECT="
>>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>>>> apigniodashboard-uat.mycompany.com"
>>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>>>>>>> REMOTE_IP="10.44.65.38"
>>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>>>> Safari/537.36"]
>>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>> POLICY
>>>>>>>> INFO
>>>>>>>>>> USER_AUTHZ
>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
>> RESOURCE="/_dash-dependencies"
>>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
>>>>>>>> APP="Ignio
>>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
>>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
>> oag_username=
>>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>>>>>>>> apigniodashboard-uat.mycompany.com
>>>>>>>>>> UserName=Joe.User@mycompantions.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
>>>>>> OS X
>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>> Chrome/88.0.4324.192
>>>>>>>>>> Safari/537.36 creationTime=1616536016811
>> maxInactiveInterval=3600000
>>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
>>>>>> Mac
>>>>>>>> OS X
>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>> Chrome/88.0.4324.192
>>>>>>>>>> Safari/537.36"] allow access to
>>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>>>>> SESSION
>>>>>>>> INFO
>>>>>>>>>> USER_SESSION
>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>>>> SUBJECT="Joe.User@mycompan'
>>>>>>>>>> *escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
>>>>>>>>>> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
>>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> SUBJECT="
>>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>>>> apigniodashboard-uat.mycompany.com"
>>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>>>>>>> REMOTE_IP="10.44.65.38"
>>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>>>> Safari/537.36"]
>>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>> POLICY
>>>>>>>> INFO
>>>>>>>>>> USER_AUTHZ
>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
>> RESOURCE="/_dash-dependencies"
>>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
>>>>>>>> APP="Ignio
>>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
>>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
>> oag_username=
>>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>>>>>>>> apigniodashboard-uat.mycompany.com
>>>>>>>>>> UserName=Joe.User@mycompantions.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
>>>>>> OS X
>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>> Chrome/88.0.4324.192
>>>>>>>>>> Safari/537.36 creationTime=1616536016811
>> maxInactiveInterval=3600000
>>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
>>>>>> Mac
>>>>>>>> OS X
>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>> Chrome/88.0.4324.192
>>>>>>>>>> Safari/537.36"] allow access to
>>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>>>>> SESSION
>>>>>>>> INFO
>>>>>>>>>> USER_SESSION
>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>>>> SUBJECT="Joe.User@mycompan'
>>>>>>>>>> *rawmsg*: '03-23T16:47:20.708-05:00
>>>>>> AUSE1OAGATST02.aws.mycompany.com
>>>>>>>>>> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
>>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> SUBJECT="
>>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>>>> apigniodashboard-uat.mycompany.com"
>>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>>>>>>> REMOTE_IP="10.44.65.38"
>>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>>>> Safari/537.36"]
>>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>> POLICY
>>>>>>>> INFO
>>>>>>>>>> USER_AUTHZ
>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
>> RESOURCE="/_dash-dependencies"
>>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
>>>>>>>> APP="Ignio
>>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
>>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
>> oag_username=
>>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>>>>>>>> apigniodashboard-uat.mycompany.com
>>>>>>>>>> UserName=Joe.User@mycompantions.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac
>>>>>> OS X
>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>> Chrome/88.0.4324.192
>>>>>>>>>> Safari/537.36 creationTime=1616536016811
>> maxInactiveInterval=3600000
>>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel
>>>>>> Mac
>>>>>>>> OS X
>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>> Chrome/88.0.4324.192
>>>>>>>>>> Safari/537.36"] allow access to
>>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>>>>> SESSION
>>>>>>>> INFO
>>>>>>>>>> USER_SESSION
>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>>>> SUBJECT="Joe.User@mycompan'
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> By inspecting the rawmsg, I can see that field four
>>>>>> (space-delimited)
>>>>>>>>>> indicates this is the ACCESS log. So if I were able to extract the
>>>>>> log
>>>>>>>>>> identifier from the msg, I could then write all access logs to the
>>>>>> same
>>>>>>>>>> daily file. There are other formats as well from the same device
>>>>>> but the
>>>>>>>>>> idea is the same.
>>>>>>>>>>
>>>>>>>>>> *Scott Slattery*
>>>>>>>>>>
>>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>>>
>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>>>
>>>>>>>>>> motorolasolutions.com
>>>>>>>>>>
>>>>>>>>>> *O: 602.529.8226*
>>>>>>>>>>
>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm> wrote:
>>>>>>>>>>
>>>>>>>>>>> the source logfile name is not included in the payload by the
>>>>>> syslog
>>>>>>>>>> spec.
>>>>>>>>>>> It
>>>>>>>>>>> may be in the case of your appliance, but we would need to see a
>>>>>> sample
>>>>>>>>>>> log to
>>>>>>>>>>> understand ho to parse it.
>>>>>>>>>>>
>>>>>>>>>>> based on your template, you are using app-name, which may be
>> listed
>>>>>>>>>>> separtely if
>>>>>>>>>>> it's a RFC5424 format log, or may be part of the syslog tag if
>>>>>> it's a
>>>>>>>>>>> RFC3164
>>>>>>>>>>> format log over the wire (neither format has a way to specify a
>>>>>> source
>>>>>>>>>> log
>>>>>>>>>>> file
>>>>>>>>>>> by default)
>>>>>>>>>>>
>>>>>>>>>>> you can look at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
>>>>>>>>>>> and see the *-cc
>>>>>>>>>>> options that you could apply to the app-name to eliminate control
>>>>>>>>>>> characters.
>>>>>>>>>>>
>>>>>>>>>>> Again, we really need to see the original log message to
>> understand
>>>>>>>>>> what's
>>>>>>>>>>> what.
>>>>>>>>>>> Please log it with the templateRSYSLOG_DebugFormat so we can see
>>>>>>>> exactly
>>>>>>>>>>> what is
>>>>>>>>>>> sent over the wire and how rsyslog has parsed it.
>>>>>>>>>>>
>>>>>>>>>>> David Lang
>>>>>>>>>>>
>>>>>>>>>>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
>>>>>>>>>>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
>>>>>>>>>>>> To: John Chivian <jchivian@chivian.com>
>>>>>>>>>>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
>>>>>>>>>>>> rsyslog-users <rsyslog@lists.adiscon.com>
>>>>>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks, John, let me try to clarify what I mean.
>>>>>>>>>>>>
>>>>>>>>>>>> Normally when I forward from a remote server to the central log
>>>>>>>>>> server, I
>>>>>>>>>>>> can include a tag that can then be used to determine the file
>>>>>> name I
>>>>>>>>>> want
>>>>>>>>>>>> on the central server. Since I have no real way to include this
>>>>>> tag
>>>>>>>>>> from
>>>>>>>>>>>> the appliance, this is not an option.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm looking for a way of inspecting the incoming packets to
>>>>>>>> determining
>>>>>>>>>>> the
>>>>>>>>>>>> source logfile name (which is included in the payload) and use
>>>>>> that
>>>>>>>>>>>> filename on the target central server. Since there are multiple
>>>>>> logs
>>>>>>>>>>> being
>>>>>>>>>>>> sent (access, audit, monitor, etc.), I'd like to segregate these
>>>>>> into
>>>>>>>>>>> their
>>>>>>>>>>>> own files. I'm already using a template with the host
>> information
>>>>>> to
>>>>>>>>>>>> dynamically create the file names. I just don't know how I can
>> go
>>>>>>>>>> beyond
>>>>>>>>>>>> this to also include the source logname.
>>>>>>>>>>>>
>>>>>>>>>>>> Here's the template I'm using. It works for all other hosts
>> where
>>>>>> I
>>>>>>>> can
>>>>>>>>>>>> configure the tag but I get garbage names from the appliance. I
>>>>>> had
>>>>>>>>>> hoped
>>>>>>>>>>>> that the appliance included some standard syslog tags but it
>>>>>> doesn't
>>>>>>>>>> seem
>>>>>>>>>>>> so.
>>>>>>>>>>>>
>>>>>>>>>>>> template(name="DynRemoteLogFile" type="string"
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
>>>>>>>>>>>>
>>>>>>>>>>>> *Scott Slattery*
>>>>>>>>>>>>
>>>>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>>>>>
>>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>>>>>
>>>>>>>>>>>> motorolasolutions.com
>>>>>>>>>>>>
>>>>>>>>>>>> *O: 602.529.8226*
>>>>>>>>>>>>
>>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <
>>>>>> jchivian@chivian.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Your use of the term “file name” is confusing. When senders
>>>>>> deliver
>>>>>>>>>> to
>>>>>>>>>>>>> rsyslog over the network there is no exchange of files or
>>>>>> filenames,
>>>>>>>>>>> only
>>>>>>>>>>>>> packets of information. Those packets are expected to be in a
>>>>>> format
>>>>>>>>>>> that
>>>>>>>>>>>>> syslog understands such that useful information (header
>> elements
>>>>>> and
>>>>>>>>>>>>> message body) may be parsed from them. If you as the rsyslog
>>>>>> admin
>>>>>>>>>>> choose
>>>>>>>>>>>>> to use some of that header information to compose filenames for
>>>>>>>> output
>>>>>>>>>>>>> files, then yes you are sort of at the mercy of the senders
>>>>>> content
>>>>>>>>>>>>> (especially if the sender doesn’t follow the syslog rules).
>>>>>> However,
>>>>>>>>>>> there
>>>>>>>>>>>>> are functions in the advanced syntax that can be used to
>> perform
>>>>>> the
>>>>>>>>>>> type
>>>>>>>>>>>>> of character replacements you’re talking about.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It is common practice to use the syslog header/rsyslog property
>>>>>>>>>> element
>>>>>>>>>>>>> called “hostname” for just such purposes. Is this what you’re
>>>>>>>> talking
>>>>>>>>>>>>> about? You’d have to provide your configuration for real
>>>>>> analysis,
>>>>>>>> at
>>>>>>>>>>>>> least the part you perceive to be responsible for the problem.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
>>>>>>>>>>>>> rsyslog@lists.adiscon.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have a configured central log collector using rsyslog. A few
>>>>>> of
>>>>>>>>>> the
>>>>>>>>>>>>>> devices forwarding their logs are appliances that have no
>>>>>>>>>>> configuration
>>>>>>>>>>>>>> options other than the IP forwarding address and protocol. I
>>>>>> cannot
>>>>>>>>>>>>> control
>>>>>>>>>>>>>> what file names are being sent.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Unfortunately, they are sending unintelligible file names with
>>>>>>>>>>> characters
>>>>>>>>>>>>>> that normally would be escaped. Is there any way I can control
>>>>>> or
>>>>>>>>>>> alter
>>>>>>>>>>>>> the
>>>>>>>>>>>>>> incoming file name to normalize it to avoid these odd
>>>>>> characters?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For example, could I establish a character map that maps the
>>>>>>>>>> unallowed
>>>>>>>>>>>>>> character to something acceptable?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Scott Slattery*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> motorolasolutions.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *O: 602.529.822*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *For more information on how and why we collect your personal
>>>>>>>>>>>>>> information, please visit our Privacy Policy
>>>>>>>>>>>>>> <
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>>>>>>>>>> .*
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
>>>>>>>>>>>>>> What's up with rsyslog? Follow
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
>>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>> by
>>>>>> a
>>>>>>>>>>> myriad
>>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>>>>> if
>>>>>>>> you
>>>>>>>>>>>>> DON'T LIKE THAT.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *For more information on how and why we collect your personal
>>>>>>>>>>>> information, please visit our Privacy Policy
>>>>>>>>>>>> <
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>>>>>>>> .*
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
>>>>>>>>>>>> What's up with rsyslog? Follow
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>> a
>>>>>>>>>> myriad
>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if
>>>>>> you
>>>>>>>>>>> DON'T LIKE THAT.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *For more information on how and why we collect your personal
>>>>>>>>>> information, please visit our Privacy Policy
>>>>>>>>>> <
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>>>>>>> .*
>>>>>>>>>> _______________________________________________
>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
>>>>>>>>>> What's up with rsyslog? Follow
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>> myriad
>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>>>>>> you
>>>>>>>>>> DON'T LIKE THAT.
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>>
>>>>>>> *For more information on how and why we collect your personal
>>>>>>> information, please visit our Privacy Policy
>>>>>>> <
>>>>>>
>>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>>> .*
>>>>>>> _______________________________________________
>>>>>>> rsyslog mailing list
>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=7hc9b2gSfvsfQzXvsRVXqGeP4Rz_tJtrykpjqkGBIdk&e=
>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=bDybhvts9KqI-WEfMHo7O40ulQK5M6Tg4tnKNRRIhE8&e=
>>>>>>> What's up with rsyslog? Follow
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=zPhpcUPlK_CWLiU75_WwjyQGz5mFTiXc1yyBIfppkrI&e=
>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>> myriad
>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>>> DON'T LIKE THAT.
>>>>>>
>>>>>
>>>>>
>>>
>>>
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
*Rsyslogd version:*
rsyslogd 8.24.0-52.amzn2, compiled with:
PLATFORM: x86_64-koji-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 64

Here is my config:
$MaxMessageSize 64k
$EscapeControlCharactersOnReceive on

module(load="imuxsock")
module(load="imjournal" StateFile="imjournal.state")
module(load="imklog")
module(load="immark")
module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat"
DirCreateMode="0705" FileCreateMode="0704")
module(load="impstats" interval="600" log.syslog="off" severity="7"
log.file="/var/log/impstats.log")

$DynaFileCacheSize 100
global(workDirectory="/var/lib/rsyslog" localHostname="cloudlogmaster01")

$MainMsgQueueSize 200000
$MainMsgQueueHighWaterMark 120000
$MainMsgQueueDiscardMark 160000
$MainMsgQueueWorkerThreads 100
$RulesetCreateMainQueue on

$imjournalRatelimitInterval 0
$imjournalRatelimitBurst 0
#### RULES ####

template(name="debug" type="string" string="Debug line with all
properties:\nFROMHOST: '%FROMHOST%'\n, FROMHOST-IP: '%fromhost-ip%'\n,
HOSTNAME: '%HOSTNAME%'\n, PRI: %PRI%,\nSYSLOGTAG '%syslogtag%'\n,
PROGRAMNAME: '%programname%'\n, APP-NAME: '%app-name%'\n, PROCID:
'%PROCID%'\n, MSGID: '%MSGID%'\n, TIMESTAMP: '%TIMESTAMP%'\n,
STRUCTURED-DATA: '%STRUCTURED-DATA%'\n, msg: '%msg%'\nRAWMSG:
'%rawmsg%',\n\nRAWMSG-AFTER-PRI: '%rawmsg-after-pri%'\nJSONMESG:
'%jsonmesg%'\n")

template(name="DynRemoteLogFile" type="string"
string="/splunklog/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")

template(name="MalformedMsgFormatter" type="string" string="%timegenerated%
%fromhost% %rawmsg:::drop-last-lf%\n")
template(name="DynRemoteMalformed" type="string"
string="/splunklog/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-oag-audit.log")

template(name="DynRemoteLogFile" type="list") {
constant(value="/splunklog/remote/")
property(name="FROMHOST")
constant(value="-")
property(name="FROMHOST-IP")
constant(value="/")
property(name="$YEAR")
constant(value="-")
property(name="$MONTH")
constant(value="-")
property(name="$DAY")
constant(value="-")
property(name="syslogtag" position.from="1" position.to="32")
constant(value="-")
property(name="app-name" controlcharacters="drop")
constant(value=".log")
}

ruleset(name="RemoteServer"){
if ($fromhost-ip startswith '10.41.102') then {
action(type="omfile" dynafile="DynRemoteMalformed"
template="MalformedMsgFormatter")
stop
}

action(type="omfile" dynaFile="DynRemoteLogFile")
}


# module imtcp
module(load="imtcp" KeepAlive="on")
input(type="imtcp" port="10514" ruleset="RemoteServer")


#kern.* /dev/console
*.info;mail.none;authpriv.none;cron.none /var/log/messages
authpriv.* /var/log/secure
mail.* /var/log/maillog
cron.* /var/log/cron
*.emerg :omusrmsg:*
uucp,news.crit /var/log/spooler
local7.* /var/log/boot.log

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: Scott.Slattery@MotorolaSolutions.com




On Thu, Mar 25, 2021 at 3:20 PM David Lang <david@lang.hm> wrote:

> what version of rsyslog are you running. can you post your full config?
>
> if you are receiving via TCP and it's not splitting the logs based on
> newlines,
> something very odd is happening.
>
> David Lang
>
> On Thu, 25 Mar 2021, Scott Slattery wrote:
>
> > Date: Thu, 25 Mar 2021 15:07:22 -0700
> > From: Scott Slattery <scott.slattery@motorolasolutions.com>
> > To: David Lang <david@lang.hm>
> > Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
> > rsyslog-users <rsyslog@lists.adiscon.com>, mariusz.kruk@safecomp.com
> > Subject: Re: [rsyslog] Altering forwarded logfile names
> >
> > So I'm looking at the pcap and on the first log record the header
> > information looks fine and the record terminated by 0x0A line feed.
> >
> > When I look at the record containing this record written to the rsyslog
> > file, the record length is 65586 (64k) long which is what I think you
> were
> > referring to earlier. Why would rsyslog write such a long record? I
> > initially thought my data may be getting truncated so I updated
> > '$MaxMessageSize 64k'. Any suggestion how I fix this behavior.
> >
> > I can confirm that each log record in the pcap is terminated by 0x0A and
> > looks in tact so must be something I need to do on rsyslog to fix this.
> >
> > *Scott Slattery*
> >
> > *Sr. Systems & Cloud Architect*
> >
> > *Cloud, Compute, Information & Architecture Team*
> >
> > motorolasolutions.com
> >
> > *O: 602.529.8226*
> >
> > *E*: Scott.Slattery@MotorolaSolutions.com
> >
> >
> >
> >
> > On Thu, Mar 25, 2021 at 1:15 PM David Lang <david@lang.hm> wrote:
> >
> >> you may want to capture with -X so that it decodes it into hex and you
> can
> >> see
> >> newlines vs linefeeds
> >>
> >> David Lang
> >>
> >> On Thu, 25 Mar 2021, Scott Slattery wrote:
> >>
> >>> Date: Thu, 25 Mar 2021 12:47:43 -0700
> >>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> >>> To: David Lang <david@lang.hm>
> >>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
> >>> rsyslog-users <rsyslog@lists.adiscon.com>,
> mariusz.kruk@safecomp.com
> >>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>
> >>> Thanks David, I just grabbed my first capture and am looking at it. I'm
> >>> using TCP vs UDP but your comment makes sense. I'll certainly share
> with
> >>> you my observations once I have some data to refer to. This has been a
> >>> learning experience. It's also the first time I've ever seen such an
> >> issue
> >>> with rsyslog so it's a bit of a puzzle.
> >>>
> >>> *Scott Slattery*
> >>>
> >>> *Sr. Systems & Cloud Architect*
> >>>
> >>> *Cloud, Compute, Information & Architecture Team*
> >>>
> >>> motorolasolutions.com
> >>>
> >>> *O: 602.529.8226*
> >>>
> >>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Mar 25, 2021 at 12:32 PM David Lang <david@lang.hm> wrote:
> >>>
> >>>> the rawmsg field in the debugformat output shows exactly what rsyslog
> is
> >>>> seeing.
> >>>>
> >>>> the reason I asked you to check multiple entries is that if rsyslog
> does
> >>>> not see
> >>>> the separator (due to either multiple messages in one UDP packet, or
> >>>> missing
> >>>> newlines in a TCP stream) it will combine what are intended as
> multiple
> >>>> messages
> >>>> together into one message as far as rsyslog is concerned, and only
> break
> >>>> it into
> >>>> multiple messages when it hits the configured maxmessagesize size.
> That
> >>>> breaks
> >>>> things at arbitrary points in the message, so as you look at multiple
> >>>> messages,
> >>>> they will not all start the same way.
> >>>>
> >>>> a tcpdump -s0 -A -n (with the appropriate port/ip filter) will also
> show
> >>>> exactly
> >>>> what's happening over the wire.
> >>>>
> >>>> David Lang
> >>>>
> >>>> On Thu, 25 Mar 2021, Scott Slattery wrote:
> >>>>
> >>>>> Date: Thu, 25 Mar 2021 12:23:03 -0700
> >>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> >>>>> To: Rainer Gerhards <rgerhards@hq.adiscon.com>
> >>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
> >> david@lang.hm
> >>>>> ,
> >>>>> mariusz.kruk@safecomp.com
> >>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>>>
> >>>>> Hey Guys, before I consume more of your time, I'm capturing the
> packets
> >>>>> from the source device to determine if any of the payload is lost on
> >> its
> >>>>> way to the rsyslog collector. After I analyze further, I'll share my
> >>>>> observations. I'm hoping nothing is getting dropped over the wire.
> >> Thanks
> >>>>> for your patience and help so far. I expect to have additional
> details
> >>>>> shortly.
> >>>>>
> >>>>> *Scott Slattery*
> >>>>>
> >>>>> *Sr. Systems & Cloud Architect*
> >>>>>
> >>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>
> >>>>> motorolasolutions.com
> >>>>>
> >>>>> *O: 602.529.8226*
> >>>>>
> >>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Mar 25, 2021 at 12:51 AM Rainer Gerhards <
> >>>> rgerhards@hq.adiscon.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Sorry if I didn't catch it, but did you post the configuration? If
> >>>>>> not, it would probably be useful.
> >>>>>>
> >>>>>> Looking at the message properties, this looks like the message was
> >>>>>> received via UDP but had a LF inside it which was intended as a
> frame
> >>>>>> separator. If so, that is another bug in the vendor's
> implementation.
> >>>>>> Framing is used for TCP, but not UDP. For UDP it is strictly one
> >>>>>> message per package.
> >>>>>>
> >>>>>> HTH,
> >>>>>> Rainer
> >>>>>>
> >>>>>> El mié, 24 mar 2021 a las 22:23, Scott Slattery via rsyslog
> >>>>>> (<rsyslog@lists.adiscon.com>) escribió:
> >>>>>>>
> >>>>>>> From what I can discern since there are lots of transactions, there
> >>>> seem
> >>>>>> to
> >>>>>>> be multiple payload formats. For all that I have so far analyzed,
> the
> >>>>>> date
> >>>>>>> truncation issue is the same regardless of the payload received.
> Like
> >>>>>> you,
> >>>>>>> I've never used mmnormalize but I will look at it as an option.
> I've
> >>>>>>> reached out to the product support team to better understand if
> there
> >>>> is
> >>>>>> a
> >>>>>>> appliance-side misconfiguration and something that can be corrected
> >>>> there
> >>>>>>> or if I need to make adjustments on the rsyslog side to clean up
> the
> >>>>>>> records.
> >>>>>>>
> >>>>>>> What I meant is that there are multiple message formats (payload
> >> data)
> >>>>>> but
> >>>>>>> there seems to be a consistent truncation of the date field just as
> >> you
> >>>>>> saw
> >>>>>>> in the app-name tag field.
> >>>>>>>
> >>>>>>> I appreciate your ongoing feedback and help. This issue is new to
> me
> >>>> and
> >>>>>>> will need a creative solution if the vendor has no answers.
> >>>>>>>
> >>>>>>> Many thanks,
> >>>>>>>
> >>>>>>>
> >>>>>>> *Scott Slattery*
> >>>>>>>
> >>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>
> >>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>
> >>>>>>> motorolasolutions.com
> >>>>>>>
> >>>>>>> *O: 602.529.8226*
> >>>>>>>
> >>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Mar 24, 2021 at 2:14 PM David Lang <david@lang.hm> wrote:
> >>>>>>>
> >>>>>>>> when you say 'each is incorrect but the same format'
> >>>>>>>>
> >>>>>>>> does that mean that every log has the year missing? or that every
> >> log
> >>>>>> is
> >>>>>>>> combining the logs together?
> >>>>>>>>
> >>>>>>>> I'll note that it's possible to define a custom parser using the
> >>>>>>>> mmnormalize
> >>>>>>>> library and add it to the parsing stack. I helped define the need
> >> for
> >>>>>> such
> >>>>>>>> a
> >>>>>>>> capability, but haven't used it yet.
> >>>>>>>>
> >>>>>>>> David Lang
> >>>>>>>>
> >>>>>>>> On Wed, 24 Mar 2021, Scott Slattery wrote:
> >>>>>>>>
> >>>>>>>>> Date: Wed, 24 Mar 2021 08:58:07 -0700
> >>>>>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> >>>>>>>>> To: mariusz.kruk@safecomp.com
> >>>>>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
> >>>>>> david@lang.hm
> >>>>>>>>>
> >>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>>>>>>>
> >>>>>>>>> Yes, all the logs are coming in this way. There are three
> separate
> >>>>>> logs
> >>>>>>>>> being sent and each is incorrect in its own right but the same
> >> format
> >>>>>>>>> behavior is consistent across each of the three logs. I will look
> >>>>>> further
> >>>>>>>>> into the suggestions you have provided and have already reached
> out
> >>>>>> to
> >>>>>>>> the
> >>>>>>>>> appliance support team and will raise the nonconformance with the
> >>>>>> RFC.
> >>>>>>>>>
> >>>>>>>>> This does provide me some valuable guidance and I thank each of
> you
> >>>>>> for
> >>>>>>>>> your feedback.
> >>>>>>>>>
> >>>>>>>>> *Scott Slattery*
> >>>>>>>>>
> >>>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>>
> >>>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>>
> >>>>>>>>> motorolasolutions.com
> >>>>>>>>>
> >>>>>>>>> *O: 602.529.8226*
> >>>>>>>>>
> >>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com>
> >> wrote:
> >>>>>>>>>
> >>>>>>>>>> There's something strange with this log entry. Notice that it
> >> looks
> >>>>>> as
> >>>>>>>> if
> >>>>>>>>>> it were more than one log entries joined with LF character
> >> (escaped
> >>>>>> on
> >>>>>>>>>> input as #012). And all "further" entries start with a - still
> >>>>>>>>>> nonconformant to the RFC but "fuller" - timestamp containing
> >>>>>> year.It's
> >>>>>>>> the
> >>>>>>>>>> leading event that seems to be trimmed in the front somehow. Do
> >> all
> >>>>>>>> events
> >>>>>>>>>> arrive like this?Wys?ano z telefonu Samsung<div>
> >>>>>>>>>> </div><div>
> >>>>>>>>>> </div><!-- originalMessage --><div>-------- Original message
> >>>>>>>>>> --------</div><div>From: Scott Slattery via rsyslog <
> >>>>>>>>>> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021 01:52
> >>>>>>>>>> (GMT+01:00) </div><div>To: David Lang <david@lang.hm>
> >>>>>> </div><div>Cc:
> >>>>>>>>>> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott
> >>>>>> Slattery
> >>>>>>>> via
> >>>>>>>>>> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re:
> >>>>>> [rsyslog]
> >>>>>>>>>> Altering forwarded logfile names </div><div>
> >>>>>>>>>> </div>
> >>>>>>>>>>
> >>>>>>>>>> Hi David, fortunately I had already done this. I'm including an
> >>>>>> actual
> >>>>>>>> log
> >>>>>>>>>> entry but have anonymized the data to keep the actual user and
> >> email
> >>>>>>>>>> address confidential:
> >>>>>>>>>>
> >>>>>>>>>> Debug line with all properties:
> >>>>>>>>>> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
> >>>>>>>>>> , fromhost-ip: '10.41.102.143'
> >>>>>>>>>> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
> >>>>>>>>>> , PRI: 13,
> >>>>>>>>>> syslogtag '03-23T16:'
> >>>>>>>>>> , programname: '03-23T16'
> >>>>>>>>>> , APP-NAME: '03-23T16'
> >>>>>>>>>> , PROCID: '-'
> >>>>>>>>>> , MSGID: '-'
> >>>>>>>>>> , TIMNESTAMP: 'Mar 23 21:47:13'
> >>>>>>>>>> , STRUCTURED-DATA: '-'
> >>>>>>>>>> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> >>>>>>>> ACCESS_GATEWAY
> >>>>>>>>>> ACCESS AUTHZ SESSION INFO USER_SESSION
> >>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >> SUBJECT="
> >>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>>>> apigniodashboard-uat.mycompany.com"
> >>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> >>>>>>>> REMOTE_IP="10.44.65.38"
> >>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>>>> Safari/537.36"]
> >>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >> POLICY
> >>>>>>>> INFO
> >>>>>>>>>> USER_AUTHZ
> >> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
> >> RESOURCE="/_dash-dependencies"
> >>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> >>>>>>>> APP="Ignio
> >>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A
> -
> >>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
> >> oag_username=
> >>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
> >>>>>>>> apigniodashboard-uat.mycompany.com
> >>>>>>>>>> UserName=Joe.User@mycompantions.com
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
> Mac
> >>>>>> OS X
> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> >> Chrome/88.0.4324.192
> >>>>>>>>>> Safari/537.36 creationTime=1616536016811
> >> maxInactiveInterval=3600000
> >>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
> Intel
> >>>>>> Mac
> >>>>>>>> OS X
> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> >> Chrome/88.0.4324.192
> >>>>>>>>>> Safari/537.36"] allow access to
> >>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>>>> SESSION
> >>>>>>>> INFO
> >>>>>>>>>> USER_SESSION
> >>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>>>> SUBJECT="Joe.User@mycompan'
> >>>>>>>>>> *escaped msg*: '47:20.708-05:00
> AUSE1OAGATST02.aws.mycompany.com
> >>>>>>>>>> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
> >>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >> SUBJECT="
> >>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>>>> apigniodashboard-uat.mycompany.com"
> >>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> >>>>>>>> REMOTE_IP="10.44.65.38"
> >>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>>>> Safari/537.36"]
> >>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >> POLICY
> >>>>>>>> INFO
> >>>>>>>>>> USER_AUTHZ
> >> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
> >> RESOURCE="/_dash-dependencies"
> >>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> >>>>>>>> APP="Ignio
> >>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A
> -
> >>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
> >> oag_username=
> >>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
> >>>>>>>> apigniodashboard-uat.mycompany.com
> >>>>>>>>>> UserName=Joe.User@mycompantions.com
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
> Mac
> >>>>>> OS X
> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> >> Chrome/88.0.4324.192
> >>>>>>>>>> Safari/537.36 creationTime=1616536016811
> >> maxInactiveInterval=3600000
> >>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
> Intel
> >>>>>> Mac
> >>>>>>>> OS X
> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> >> Chrome/88.0.4324.192
> >>>>>>>>>> Safari/537.36"] allow access to
> >>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>>>> SESSION
> >>>>>>>> INFO
> >>>>>>>>>> USER_SESSION
> >>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>>>> SUBJECT="Joe.User@mycompan'
> >>>>>>>>>> *rawmsg*: '03-23T16:47:20.708-05:00
> >>>>>> AUSE1OAGATST02.aws.mycompany.com
> >>>>>>>>>> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
> >>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >> SUBJECT="
> >>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>>>> apigniodashboard-uat.mycompany.com"
> >>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> >>>>>>>> REMOTE_IP="10.44.65.38"
> >>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>>>> Safari/537.36"]
> >>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >> POLICY
> >>>>>>>> INFO
> >>>>>>>>>> USER_AUTHZ
> >> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
> >> RESOURCE="/_dash-dependencies"
> >>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
> >>>>>>>> APP="Ignio
> >>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A
> -
> >>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
> >> oag_username=
> >>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
> >>>>>>>> apigniodashboard-uat.mycompany.com
> >>>>>>>>>> UserName=Joe.User@mycompantions.com
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
> Mac
> >>>>>> OS X
> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> >> Chrome/88.0.4324.192
> >>>>>>>>>> Safari/537.36 creationTime=1616536016811
> >> maxInactiveInterval=3600000
> >>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
> Intel
> >>>>>> Mac
> >>>>>>>> OS X
> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> >> Chrome/88.0.4324.192
> >>>>>>>>>> Safari/537.36"] allow access to
> >>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>>>> SESSION
> >>>>>>>> INFO
> >>>>>>>>>> USER_SESSION
> >>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>>>> SUBJECT="Joe.User@mycompan'
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> By inspecting the rawmsg, I can see that field four
> >>>>>> (space-delimited)
> >>>>>>>>>> indicates this is the ACCESS log. So if I were able to extract
> the
> >>>>>> log
> >>>>>>>>>> identifier from the msg, I could then write all access logs to
> the
> >>>>>> same
> >>>>>>>>>> daily file. There are other formats as well from the same device
> >>>>>> but the
> >>>>>>>>>> idea is the same.
> >>>>>>>>>>
> >>>>>>>>>> *Scott Slattery*
> >>>>>>>>>>
> >>>>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>>>
> >>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>>>
> >>>>>>>>>> motorolasolutions.com
> >>>>>>>>>>
> >>>>>>>>>> *O: 602.529.8226*
> >>>>>>>>>>
> >>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm>
> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> the source logfile name is not included in the payload by the
> >>>>>> syslog
> >>>>>>>>>> spec.
> >>>>>>>>>>> It
> >>>>>>>>>>> may be in the case of your appliance, but we would need to see
> a
> >>>>>> sample
> >>>>>>>>>>> log to
> >>>>>>>>>>> understand ho to parse it.
> >>>>>>>>>>>
> >>>>>>>>>>> based on your template, you are using app-name, which may be
> >> listed
> >>>>>>>>>>> separtely if
> >>>>>>>>>>> it's a RFC5424 format log, or may be part of the syslog tag if
> >>>>>> it's a
> >>>>>>>>>>> RFC3164
> >>>>>>>>>>> format log over the wire (neither format has a way to specify a
> >>>>>> source
> >>>>>>>>>> log
> >>>>>>>>>>> file
> >>>>>>>>>>> by default)
> >>>>>>>>>>>
> >>>>>>>>>>> you can look at
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
> >>>>>>>>>>> and see the *-cc
> >>>>>>>>>>> options that you could apply to the app-name to eliminate
> control
> >>>>>>>>>>> characters.
> >>>>>>>>>>>
> >>>>>>>>>>> Again, we really need to see the original log message to
> >> understand
> >>>>>>>>>> what's
> >>>>>>>>>>> what.
> >>>>>>>>>>> Please log it with the templateRSYSLOG_DebugFormat so we can
> see
> >>>>>>>> exactly
> >>>>>>>>>>> what is
> >>>>>>>>>>> sent over the wire and how rsyslog has parsed it.
> >>>>>>>>>>>
> >>>>>>>>>>> David Lang
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
> >>>>>>>>>>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
> >>>>>>>>>>>> To: John Chivian <jchivian@chivian.com>
> >>>>>>>>>>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
> >>>>>>>>>>>> rsyslog-users <rsyslog@lists.adiscon.com>
> >>>>>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks, John, let me try to clarify what I mean.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Normally when I forward from a remote server to the central
> log
> >>>>>>>>>> server, I
> >>>>>>>>>>>> can include a tag that can then be used to determine the file
> >>>>>> name I
> >>>>>>>>>> want
> >>>>>>>>>>>> on the central server. Since I have no real way to include
> this
> >>>>>> tag
> >>>>>>>>>> from
> >>>>>>>>>>>> the appliance, this is not an option.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'm looking for a way of inspecting the incoming packets to
> >>>>>>>> determining
> >>>>>>>>>>> the
> >>>>>>>>>>>> source logfile name (which is included in the payload) and use
> >>>>>> that
> >>>>>>>>>>>> filename on the target central server. Since there are
> multiple
> >>>>>> logs
> >>>>>>>>>>> being
> >>>>>>>>>>>> sent (access, audit, monitor, etc.), I'd like to segregate
> these
> >>>>>> into
> >>>>>>>>>>> their
> >>>>>>>>>>>> own files. I'm already using a template with the host
> >> information
> >>>>>> to
> >>>>>>>>>>>> dynamically create the file names. I just don't know how I can
> >> go
> >>>>>>>>>> beyond
> >>>>>>>>>>>> this to also include the source logname.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Here's the template I'm using. It works for all other hosts
> >> where
> >>>>>> I
> >>>>>>>> can
> >>>>>>>>>>>> configure the tag but I get garbage names from the appliance.
> I
> >>>>>> had
> >>>>>>>>>> hoped
> >>>>>>>>>>>> that the appliance included some standard syslog tags but it
> >>>>>> doesn't
> >>>>>>>>>> seem
> >>>>>>>>>>>> so.
> >>>>>>>>>>>>
> >>>>>>>>>>>> template(name="DynRemoteLogFile" type="string"
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Scott Slattery*
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>>>>>
> >>>>>>>>>>>> motorolasolutions.com
> >>>>>>>>>>>>
> >>>>>>>>>>>> *O: 602.529.8226*
> >>>>>>>>>>>>
> >>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <
> >>>>>> jchivian@chivian.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Your use of the term “file name” is confusing. When senders
> >>>>>> deliver
> >>>>>>>>>> to
> >>>>>>>>>>>>> rsyslog over the network there is no exchange of files or
> >>>>>> filenames,
> >>>>>>>>>>> only
> >>>>>>>>>>>>> packets of information. Those packets are expected to be in
> a
> >>>>>> format
> >>>>>>>>>>> that
> >>>>>>>>>>>>> syslog understands such that useful information (header
> >> elements
> >>>>>> and
> >>>>>>>>>>>>> message body) may be parsed from them. If you as the rsyslog
> >>>>>> admin
> >>>>>>>>>>> choose
> >>>>>>>>>>>>> to use some of that header information to compose filenames
> for
> >>>>>>>> output
> >>>>>>>>>>>>> files, then yes you are sort of at the mercy of the senders
> >>>>>> content
> >>>>>>>>>>>>> (especially if the sender doesn’t follow the syslog rules).
> >>>>>> However,
> >>>>>>>>>>> there
> >>>>>>>>>>>>> are functions in the advanced syntax that can be used to
> >> perform
> >>>>>> the
> >>>>>>>>>>> type
> >>>>>>>>>>>>> of character replacements you’re talking about.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> It is common practice to use the syslog header/rsyslog
> property
> >>>>>>>>>> element
> >>>>>>>>>>>>> called “hostname” for just such purposes. Is this what
> you’re
> >>>>>>>> talking
> >>>>>>>>>>>>> about? You’d have to provide your configuration for real
> >>>>>> analysis,
> >>>>>>>> at
> >>>>>>>>>>>>> least the part you perceive to be responsible for the
> problem.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
> >>>>>>>>>>>>> rsyslog@lists.adiscon.com> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I have a configured central log collector using rsyslog. A
> few
> >>>>>> of
> >>>>>>>>>> the
> >>>>>>>>>>>>>> devices forwarding their logs are appliances that have no
> >>>>>>>>>>> configuration
> >>>>>>>>>>>>>> options other than the IP forwarding address and protocol. I
> >>>>>> cannot
> >>>>>>>>>>>>> control
> >>>>>>>>>>>>>> what file names are being sent.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Unfortunately, they are sending unintelligible file names
> with
> >>>>>>>>>>> characters
> >>>>>>>>>>>>>> that normally would be escaped. Is there any way I can
> control
> >>>>>> or
> >>>>>>>>>>> alter
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>> incoming file name to normalize it to avoid these odd
> >>>>>> characters?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> For example, could I establish a character map that maps the
> >>>>>>>>>> unallowed
> >>>>>>>>>>>>>> character to something acceptable?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> thanks,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *Scott Slattery*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> motorolasolutions.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *O: 602.529.822*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *For more information on how and why we collect your
> personal
> >>>>>>>>>>>>>> information, please visit our Privacy Policy
> >>>>>>>>>>>>>> <
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>>>>>>>>> .*
> >>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>> rsyslog mailing list
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
> >>>>>>>>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
> >>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> >> by
> >>>>>> a
> >>>>>>>>>>> myriad
> >>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> POST
> >>>>>> if
> >>>>>>>> you
> >>>>>>>>>>>>> DON'T LIKE THAT.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> *For more information on how and why we collect your personal
> >>>>>>>>>>>> information, please visit our Privacy Policy
> >>>>>>>>>>>> <
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>>>>>>> .*
> >>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>> rsyslog mailing list
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
> >>>>>>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
> >>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> by
> >> a
> >>>>>>>>>> myriad
> >>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> >> if
> >>>>>> you
> >>>>>>>>>>> DON'T LIKE THAT.
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> *For more information on how and why we collect your personal
> >>>>>>>>>> information, please visit our Privacy Policy
> >>>>>>>>>> <
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>>>>>> .*
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>> rsyslog mailing list
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
> >>>>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
> >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
> a
> >>>>>> myriad
> >>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if
> >>>>>> you
> >>>>>>>>>> DON'T LIKE THAT.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>>
> >>>>>>>
> >>>>>>> *For more information on how and why we collect your personal
> >>>>>>> information, please visit our Privacy Policy
> >>>>>>> <
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>> .*
> >>>>>>> _______________________________________________
> >>>>>>> rsyslog mailing list
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=7hc9b2gSfvsfQzXvsRVXqGeP4Rz_tJtrykpjqkGBIdk&e=
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=bDybhvts9KqI-WEfMHo7O40ulQK5M6Tg4tnKNRRIhE8&e=
> >>>>>>> What's up with rsyslog? Follow
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=zPhpcUPlK_CWLiU75_WwjyQGz5mFTiXc1yyBIfppkrI&e=
> >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>>> myriad
> >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> you
> >>>>>> DON'T LIKE THAT.
> >>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >
> >

--


*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
Upon further review, I'm seeing packet lengths from 4434 down to 111. It
appears once the packet length exceeds 2919 the trailing 0x0a is missing
from any remaining packets larger than 2919.

If I sort by packet size, it jumps from 2919 to 2974 then all further line
feed character is missing.

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: Scott.Slattery@MotorolaSolutions.com




On Thu, Mar 25, 2021 at 3:24 PM Scott Slattery <
scott.slattery@motorolasolutions.com> wrote:

>
> *Rsyslogd version:*
> rsyslogd 8.24.0-52.amzn2, compiled with:
> PLATFORM: x86_64-koji-linux-gnu
> PLATFORM (lsb_release -d):
> FEATURE_REGEXP: Yes
> GSSAPI Kerberos 5 support: Yes
> FEATURE_DEBUG (debug build, slow code): No
> 32bit Atomic operations supported: Yes
> 64bit Atomic operations supported: Yes
> memory allocator: system default
> Runtime Instrumentation (slow code): No
> uuid support: Yes
> Number of Bits in RainerScript integers: 64
>
> Here is my config:
> $MaxMessageSize 64k
> $EscapeControlCharactersOnReceive on
>
> module(load="imuxsock")
> module(load="imjournal" StateFile="imjournal.state")
> module(load="imklog")
> module(load="immark")
> module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat"
> DirCreateMode="0705" FileCreateMode="0704")
> module(load="impstats" interval="600" log.syslog="off" severity="7"
> log.file="/var/log/impstats.log")
>
> $DynaFileCacheSize 100
> global(workDirectory="/var/lib/rsyslog" localHostname="cloudlogmaster01")
>
> $MainMsgQueueSize 200000
> $MainMsgQueueHighWaterMark 120000
> $MainMsgQueueDiscardMark 160000
> $MainMsgQueueWorkerThreads 100
> $RulesetCreateMainQueue on
>
> $imjournalRatelimitInterval 0
> $imjournalRatelimitBurst 0
> #### RULES ####
>
> template(name="debug" type="string" string="Debug line with all
> properties:\nFROMHOST: '%FROMHOST%'\n, FROMHOST-IP: '%fromhost-ip%'\n,
> HOSTNAME: '%HOSTNAME%'\n, PRI: %PRI%,\nSYSLOGTAG '%syslogtag%'\n,
> PROGRAMNAME: '%programname%'\n, APP-NAME: '%app-name%'\n, PROCID:
> '%PROCID%'\n, MSGID: '%MSGID%'\n, TIMESTAMP: '%TIMESTAMP%'\n,
> STRUCTURED-DATA: '%STRUCTURED-DATA%'\n, msg: '%msg%'\nRAWMSG:
> '%rawmsg%',\n\nRAWMSG-AFTER-PRI: '%rawmsg-after-pri%'\nJSONMESG:
> '%jsonmesg%'\n")
>
> template(name="DynRemoteLogFile" type="string"
> string="/splunklog/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
>
> template(name="MalformedMsgFormatter" type="string"
> string="%timegenerated% %fromhost% %rawmsg:::drop-last-lf%\n")
> template(name="DynRemoteMalformed" type="string"
> string="/splunklog/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-oag-audit.log")
>
> template(name="DynRemoteLogFile" type="list") {
> constant(value="/splunklog/remote/")
> property(name="FROMHOST")
> constant(value="-")
> property(name="FROMHOST-IP")
> constant(value="/")
> property(name="$YEAR")
> constant(value="-")
> property(name="$MONTH")
> constant(value="-")
> property(name="$DAY")
> constant(value="-")
> property(name="syslogtag" position.from="1" position.to="32")
> constant(value="-")
> property(name="app-name" controlcharacters="drop")
> constant(value=".log")
> }
>
> ruleset(name="RemoteServer"){
> if ($fromhost-ip startswith '10.41.102') then {
> action(type="omfile" dynafile="DynRemoteMalformed"
> template="MalformedMsgFormatter")
> stop
> }
>
> action(type="omfile" dynaFile="DynRemoteLogFile")
> }
>
>
> # module imtcp
> module(load="imtcp" KeepAlive="on")
> input(type="imtcp" port="10514" ruleset="RemoteServer")
>
>
> #kern.* /dev/console
> *.info;mail.none;authpriv.none;cron.none /var/log/messages
> authpriv.* /var/log/secure
> mail.* /var/log/maillog
> cron.* /var/log/cron
> *.emerg :omusrmsg:*
> uucp,news.crit /var/log/spooler
> local7.* /var/log/boot.log
>
> *Scott Slattery*
>
> *Sr. Systems & Cloud Architect*
>
> *Cloud, Compute, Information & Architecture Team*
>
> motorolasolutions.com
>
> *O: 602.529.8226*
>
> *E*: Scott.Slattery@MotorolaSolutions.com
>
>
>
>
> On Thu, Mar 25, 2021 at 3:20 PM David Lang <david@lang.hm> wrote:
>
>> what version of rsyslog are you running. can you post your full config?
>>
>> if you are receiving via TCP and it's not splitting the logs based on
>> newlines,
>> something very odd is happening.
>>
>> David Lang
>>
>> On Thu, 25 Mar 2021, Scott Slattery wrote:
>>
>> > Date: Thu, 25 Mar 2021 15:07:22 -0700
>> > From: Scott Slattery <scott.slattery@motorolasolutions.com>
>> > To: David Lang <david@lang.hm>
>> > Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
>> > rsyslog-users <rsyslog@lists.adiscon.com>,
>> mariusz.kruk@safecomp.com
>> > Subject: Re: [rsyslog] Altering forwarded logfile names
>> >
>> > So I'm looking at the pcap and on the first log record the header
>> > information looks fine and the record terminated by 0x0A line feed.
>> >
>> > When I look at the record containing this record written to the rsyslog
>> > file, the record length is 65586 (64k) long which is what I think you
>> were
>> > referring to earlier. Why would rsyslog write such a long record? I
>> > initially thought my data may be getting truncated so I updated
>> > '$MaxMessageSize 64k'. Any suggestion how I fix this behavior.
>> >
>> > I can confirm that each log record in the pcap is terminated by 0x0A and
>> > looks in tact so must be something I need to do on rsyslog to fix this.
>> >
>> > *Scott Slattery*
>> >
>> > *Sr. Systems & Cloud Architect*
>> >
>> > *Cloud, Compute, Information & Architecture Team*
>> >
>> > motorolasolutions.com
>> >
>> > *O: 602.529.8226*
>> >
>> > *E*: Scott.Slattery@MotorolaSolutions.com
>> >
>> >
>> >
>> >
>> > On Thu, Mar 25, 2021 at 1:15 PM David Lang <david@lang.hm> wrote:
>> >
>> >> you may want to capture with -X so that it decodes it into hex and you
>> can
>> >> see
>> >> newlines vs linefeeds
>> >>
>> >> David Lang
>> >>
>> >> On Thu, 25 Mar 2021, Scott Slattery wrote:
>> >>
>> >>> Date: Thu, 25 Mar 2021 12:47:43 -0700
>> >>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>> >>> To: David Lang <david@lang.hm>
>> >>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
>> >>> rsyslog-users <rsyslog@lists.adiscon.com>,
>> mariusz.kruk@safecomp.com
>> >>> Subject: Re: [rsyslog] Altering forwarded logfile names
>> >>>
>> >>> Thanks David, I just grabbed my first capture and am looking at it.
>> I'm
>> >>> using TCP vs UDP but your comment makes sense. I'll certainly share
>> with
>> >>> you my observations once I have some data to refer to. This has been a
>> >>> learning experience. It's also the first time I've ever seen such an
>> >> issue
>> >>> with rsyslog so it's a bit of a puzzle.
>> >>>
>> >>> *Scott Slattery*
>> >>>
>> >>> *Sr. Systems & Cloud Architect*
>> >>>
>> >>> *Cloud, Compute, Information & Architecture Team*
>> >>>
>> >>> motorolasolutions.com
>> >>>
>> >>> *O: 602.529.8226*
>> >>>
>> >>> *E*: Scott.Slattery@MotorolaSolutions.com
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Mar 25, 2021 at 12:32 PM David Lang <david@lang.hm> wrote:
>> >>>
>> >>>> the rawmsg field in the debugformat output shows exactly what
>> rsyslog is
>> >>>> seeing.
>> >>>>
>> >>>> the reason I asked you to check multiple entries is that if rsyslog
>> does
>> >>>> not see
>> >>>> the separator (due to either multiple messages in one UDP packet, or
>> >>>> missing
>> >>>> newlines in a TCP stream) it will combine what are intended as
>> multiple
>> >>>> messages
>> >>>> together into one message as far as rsyslog is concerned, and only
>> break
>> >>>> it into
>> >>>> multiple messages when it hits the configured maxmessagesize size.
>> That
>> >>>> breaks
>> >>>> things at arbitrary points in the message, so as you look at multiple
>> >>>> messages,
>> >>>> they will not all start the same way.
>> >>>>
>> >>>> a tcpdump -s0 -A -n (with the appropriate port/ip filter) will also
>> show
>> >>>> exactly
>> >>>> what's happening over the wire.
>> >>>>
>> >>>> David Lang
>> >>>>
>> >>>> On Thu, 25 Mar 2021, Scott Slattery wrote:
>> >>>>
>> >>>>> Date: Thu, 25 Mar 2021 12:23:03 -0700
>> >>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>> >>>>> To: Rainer Gerhards <rgerhards@hq.adiscon.com>
>> >>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
>> >> david@lang.hm
>> >>>>> ,
>> >>>>> mariusz.kruk@safecomp.com
>> >>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>> >>>>>
>> >>>>> Hey Guys, before I consume more of your time, I'm capturing the
>> packets
>> >>>>> from the source device to determine if any of the payload is lost on
>> >> its
>> >>>>> way to the rsyslog collector. After I analyze further, I'll share my
>> >>>>> observations. I'm hoping nothing is getting dropped over the wire.
>> >> Thanks
>> >>>>> for your patience and help so far. I expect to have additional
>> details
>> >>>>> shortly.
>> >>>>>
>> >>>>> *Scott Slattery*
>> >>>>>
>> >>>>> *Sr. Systems & Cloud Architect*
>> >>>>>
>> >>>>> *Cloud, Compute, Information & Architecture Team*
>> >>>>>
>> >>>>> motorolasolutions.com
>> >>>>>
>> >>>>> *O: 602.529.8226*
>> >>>>>
>> >>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Thu, Mar 25, 2021 at 12:51 AM Rainer Gerhards <
>> >>>> rgerhards@hq.adiscon.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Sorry if I didn't catch it, but did you post the configuration? If
>> >>>>>> not, it would probably be useful.
>> >>>>>>
>> >>>>>> Looking at the message properties, this looks like the message was
>> >>>>>> received via UDP but had a LF inside it which was intended as a
>> frame
>> >>>>>> separator. If so, that is another bug in the vendor's
>> implementation.
>> >>>>>> Framing is used for TCP, but not UDP. For UDP it is strictly one
>> >>>>>> message per package.
>> >>>>>>
>> >>>>>> HTH,
>> >>>>>> Rainer
>> >>>>>>
>> >>>>>> El mié, 24 mar 2021 a las 22:23, Scott Slattery via rsyslog
>> >>>>>> (<rsyslog@lists.adiscon.com>) escribió:
>> >>>>>>>
>> >>>>>>> From what I can discern since there are lots of transactions,
>> there
>> >>>> seem
>> >>>>>> to
>> >>>>>>> be multiple payload formats. For all that I have so far analyzed,
>> the
>> >>>>>> date
>> >>>>>>> truncation issue is the same regardless of the payload received.
>> Like
>> >>>>>> you,
>> >>>>>>> I've never used mmnormalize but I will look at it as an option.
>> I've
>> >>>>>>> reached out to the product support team to better understand if
>> there
>> >>>> is
>> >>>>>> a
>> >>>>>>> appliance-side misconfiguration and something that can be
>> corrected
>> >>>> there
>> >>>>>>> or if I need to make adjustments on the rsyslog side to clean up
>> the
>> >>>>>>> records.
>> >>>>>>>
>> >>>>>>> What I meant is that there are multiple message formats (payload
>> >> data)
>> >>>>>> but
>> >>>>>>> there seems to be a consistent truncation of the date field just
>> as
>> >> you
>> >>>>>> saw
>> >>>>>>> in the app-name tag field.
>> >>>>>>>
>> >>>>>>> I appreciate your ongoing feedback and help. This issue is new to
>> me
>> >>>> and
>> >>>>>>> will need a creative solution if the vendor has no answers.
>> >>>>>>>
>> >>>>>>> Many thanks,
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> *Scott Slattery*
>> >>>>>>>
>> >>>>>>> *Sr. Systems & Cloud Architect*
>> >>>>>>>
>> >>>>>>> *Cloud, Compute, Information & Architecture Team*
>> >>>>>>>
>> >>>>>>> motorolasolutions.com
>> >>>>>>>
>> >>>>>>> *O: 602.529.8226*
>> >>>>>>>
>> >>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Wed, Mar 24, 2021 at 2:14 PM David Lang <david@lang.hm> wrote:
>> >>>>>>>
>> >>>>>>>> when you say 'each is incorrect but the same format'
>> >>>>>>>>
>> >>>>>>>> does that mean that every log has the year missing? or that every
>> >> log
>> >>>>>> is
>> >>>>>>>> combining the logs together?
>> >>>>>>>>
>> >>>>>>>> I'll note that it's possible to define a custom parser using the
>> >>>>>>>> mmnormalize
>> >>>>>>>> library and add it to the parsing stack. I helped define the need
>> >> for
>> >>>>>> such
>> >>>>>>>> a
>> >>>>>>>> capability, but haven't used it yet.
>> >>>>>>>>
>> >>>>>>>> David Lang
>> >>>>>>>>
>> >>>>>>>> On Wed, 24 Mar 2021, Scott Slattery wrote:
>> >>>>>>>>
>> >>>>>>>>> Date: Wed, 24 Mar 2021 08:58:07 -0700
>> >>>>>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>> >>>>>>>>> To: mariusz.kruk@safecomp.com
>> >>>>>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
>> >>>>>> david@lang.hm
>> >>>>>>>>>
>> >>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>> >>>>>>>>>
>> >>>>>>>>> Yes, all the logs are coming in this way. There are three
>> separate
>> >>>>>> logs
>> >>>>>>>>> being sent and each is incorrect in its own right but the same
>> >> format
>> >>>>>>>>> behavior is consistent across each of the three logs. I will
>> look
>> >>>>>> further
>> >>>>>>>>> into the suggestions you have provided and have already reached
>> out
>> >>>>>> to
>> >>>>>>>> the
>> >>>>>>>>> appliance support team and will raise the nonconformance with
>> the
>> >>>>>> RFC.
>> >>>>>>>>>
>> >>>>>>>>> This does provide me some valuable guidance and I thank each of
>> you
>> >>>>>> for
>> >>>>>>>>> your feedback.
>> >>>>>>>>>
>> >>>>>>>>> *Scott Slattery*
>> >>>>>>>>>
>> >>>>>>>>> *Sr. Systems & Cloud Architect*
>> >>>>>>>>>
>> >>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>> >>>>>>>>>
>> >>>>>>>>> motorolasolutions.com
>> >>>>>>>>>
>> >>>>>>>>> *O: 602.529.8226*
>> >>>>>>>>>
>> >>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com>
>> >> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> There's something strange with this log entry. Notice that it
>> >> looks
>> >>>>>> as
>> >>>>>>>> if
>> >>>>>>>>>> it were more than one log entries joined with LF character
>> >> (escaped
>> >>>>>> on
>> >>>>>>>>>> input as #012). And all "further" entries start with a - still
>> >>>>>>>>>> nonconformant to the RFC but "fuller" - timestamp containing
>> >>>>>> year.It's
>> >>>>>>>> the
>> >>>>>>>>>> leading event that seems to be trimmed in the front somehow. Do
>> >> all
>> >>>>>>>> events
>> >>>>>>>>>> arrive like this?Wys?ano z telefonu Samsung<div>
>> >>>>>>>>>> </div><div>
>> >>>>>>>>>> </div><!-- originalMessage --><div>-------- Original message
>> >>>>>>>>>> --------</div><div>From: Scott Slattery via rsyslog <
>> >>>>>>>>>> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021 01:52
>> >>>>>>>>>> (GMT+01:00) </div><div>To: David Lang <david@lang.hm>
>> >>>>>> </div><div>Cc:
>> >>>>>>>>>> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott
>> >>>>>> Slattery
>> >>>>>>>> via
>> >>>>>>>>>> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re:
>> >>>>>> [rsyslog]
>> >>>>>>>>>> Altering forwarded logfile names </div><div>
>> >>>>>>>>>> </div>
>> >>>>>>>>>>
>> >>>>>>>>>> Hi David, fortunately I had already done this. I'm including an
>> >>>>>> actual
>> >>>>>>>> log
>> >>>>>>>>>> entry but have anonymized the data to keep the actual user and
>> >> email
>> >>>>>>>>>> address confidential:
>> >>>>>>>>>>
>> >>>>>>>>>> Debug line with all properties:
>> >>>>>>>>>> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
>> >>>>>>>>>> , fromhost-ip: '10.41.102.143'
>> >>>>>>>>>> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
>> >>>>>>>>>> , PRI: 13,
>> >>>>>>>>>> syslogtag '03-23T16:'
>> >>>>>>>>>> , programname: '03-23T16'
>> >>>>>>>>>> , APP-NAME: '03-23T16'
>> >>>>>>>>>> , PROCID: '-'
>> >>>>>>>>>> , MSGID: '-'
>> >>>>>>>>>> , TIMNESTAMP: 'Mar 23 21:47:13'
>> >>>>>>>>>> , STRUCTURED-DATA: '-'
>> >>>>>>>>>> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
>> >>>>>>>> ACCESS_GATEWAY
>> >>>>>>>>>> ACCESS AUTHZ SESSION INFO USER_SESSION
>> >>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> >> SUBJECT="
>> >>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>> >>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>> >>>>>>>>>> apigniodashboard-uat.mycompany.com"
>> >>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>> >>>>>>>> REMOTE_IP="10.44.65.38"
>> >>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>> >>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>> >>>>>>>> Safari/537.36"]
>> >>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>> >>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>> >> POLICY
>> >>>>>>>> INFO
>> >>>>>>>>>> USER_AUTHZ
>> >> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> >>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
>> >> RESOURCE="/_dash-dependencies"
>> >>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
>> >>>>>>>> APP="Ignio
>> >>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>> >>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW"
>> REASON="N/A -
>> >>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
>> >> oag_username=
>> >>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>> >>>>>>>> apigniodashboard-uat.mycompany.com
>> >>>>>>>>>> UserName=Joe.User@mycompantions.com
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>> >>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
>> Mac
>> >>>>>> OS X
>> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>> >> Chrome/88.0.4324.192
>> >>>>>>>>>> Safari/537.36 creationTime=1616536016811
>> >> maxInactiveInterval=3600000
>> >>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>> >>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
>> Intel
>> >>>>>> Mac
>> >>>>>>>> OS X
>> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>> >> Chrome/88.0.4324.192
>> >>>>>>>>>> Safari/537.36"] allow access to
>> >>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>> >>>>>> SESSION
>> >>>>>>>> INFO
>> >>>>>>>>>> USER_SESSION
>> >>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> >>>>>>>>>> SUBJECT="Joe.User@mycompan'
>> >>>>>>>>>> *escaped msg*: '47:20.708-05:00
>> AUSE1OAGATST02.aws.mycompany.com
>> >>>>>>>>>> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
>> >>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> >> SUBJECT="
>> >>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>> >>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>> >>>>>>>>>> apigniodashboard-uat.mycompany.com"
>> >>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>> >>>>>>>> REMOTE_IP="10.44.65.38"
>> >>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>> >>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>> >>>>>>>> Safari/537.36"]
>> >>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>> >>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>> >> POLICY
>> >>>>>>>> INFO
>> >>>>>>>>>> USER_AUTHZ
>> >> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> >>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
>> >> RESOURCE="/_dash-dependencies"
>> >>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
>> >>>>>>>> APP="Ignio
>> >>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>> >>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW"
>> REASON="N/A -
>> >>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
>> >> oag_username=
>> >>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>> >>>>>>>> apigniodashboard-uat.mycompany.com
>> >>>>>>>>>> UserName=Joe.User@mycompantions.com
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>> >>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
>> Mac
>> >>>>>> OS X
>> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>> >> Chrome/88.0.4324.192
>> >>>>>>>>>> Safari/537.36 creationTime=1616536016811
>> >> maxInactiveInterval=3600000
>> >>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>> >>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
>> Intel
>> >>>>>> Mac
>> >>>>>>>> OS X
>> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>> >> Chrome/88.0.4324.192
>> >>>>>>>>>> Safari/537.36"] allow access to
>> >>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>> >>>>>> SESSION
>> >>>>>>>> INFO
>> >>>>>>>>>> USER_SESSION
>> >>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> >>>>>>>>>> SUBJECT="Joe.User@mycompan'
>> >>>>>>>>>> *rawmsg*: '03-23T16:47:20.708-05:00
>> >>>>>> AUSE1OAGATST02.aws.mycompany.com
>> >>>>>>>>>> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
>> >>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> >> SUBJECT="
>> >>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>> >>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>> >>>>>>>>>> apigniodashboard-uat.mycompany.com"
>> >>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>> >>>>>>>> REMOTE_IP="10.44.65.38"
>> >>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>> >>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>> >>>>>>>> Safari/537.36"]
>> >>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>> >>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>> >> POLICY
>> >>>>>>>> INFO
>> >>>>>>>>>> USER_AUTHZ
>> >> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> >>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
>> >> RESOURCE="/_dash-dependencies"
>> >>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
>> >>>>>>>> APP="Ignio
>> >>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>> >>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW"
>> REASON="N/A -
>> >>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
>> >> oag_username=
>> >>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>> >>>>>>>> apigniodashboard-uat.mycompany.com
>> >>>>>>>>>> UserName=Joe.User@mycompantions.com
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>> >>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
>> Mac
>> >>>>>> OS X
>> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>> >> Chrome/88.0.4324.192
>> >>>>>>>>>> Safari/537.36 creationTime=1616536016811
>> >> maxInactiveInterval=3600000
>> >>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>> >>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
>> Intel
>> >>>>>> Mac
>> >>>>>>>> OS X
>> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>> >> Chrome/88.0.4324.192
>> >>>>>>>>>> Safari/537.36"] allow access to
>> >>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>> >>>>>> SESSION
>> >>>>>>>> INFO
>> >>>>>>>>>> USER_SESSION
>> >>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>> >>>>>>>>>> SUBJECT="Joe.User@mycompan'
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> By inspecting the rawmsg, I can see that field four
>> >>>>>> (space-delimited)
>> >>>>>>>>>> indicates this is the ACCESS log. So if I were able to extract
>> the
>> >>>>>> log
>> >>>>>>>>>> identifier from the msg, I could then write all access logs to
>> the
>> >>>>>> same
>> >>>>>>>>>> daily file. There are other formats as well from the same
>> device
>> >>>>>> but the
>> >>>>>>>>>> idea is the same.
>> >>>>>>>>>>
>> >>>>>>>>>> *Scott Slattery*
>> >>>>>>>>>>
>> >>>>>>>>>> *Sr. Systems & Cloud Architect*
>> >>>>>>>>>>
>> >>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>> >>>>>>>>>>
>> >>>>>>>>>> motorolasolutions.com
>> >>>>>>>>>>
>> >>>>>>>>>> *O: 602.529.8226*
>> >>>>>>>>>>
>> >>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm>
>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> the source logfile name is not included in the payload by the
>> >>>>>> syslog
>> >>>>>>>>>> spec.
>> >>>>>>>>>>> It
>> >>>>>>>>>>> may be in the case of your appliance, but we would need to
>> see a
>> >>>>>> sample
>> >>>>>>>>>>> log to
>> >>>>>>>>>>> understand ho to parse it.
>> >>>>>>>>>>>
>> >>>>>>>>>>> based on your template, you are using app-name, which may be
>> >> listed
>> >>>>>>>>>>> separtely if
>> >>>>>>>>>>> it's a RFC5424 format log, or may be part of the syslog tag if
>> >>>>>> it's a
>> >>>>>>>>>>> RFC3164
>> >>>>>>>>>>> format log over the wire (neither format has a way to specify
>> a
>> >>>>>> source
>> >>>>>>>>>> log
>> >>>>>>>>>>> file
>> >>>>>>>>>>> by default)
>> >>>>>>>>>>>
>> >>>>>>>>>>> you can look at
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
>> >>>>>>>>>>> and see the *-cc
>> >>>>>>>>>>> options that you could apply to the app-name to eliminate
>> control
>> >>>>>>>>>>> characters.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Again, we really need to see the original log message to
>> >> understand
>> >>>>>>>>>> what's
>> >>>>>>>>>>> what.
>> >>>>>>>>>>> Please log it with the templateRSYSLOG_DebugFormat so we can
>> see
>> >>>>>>>> exactly
>> >>>>>>>>>>> what is
>> >>>>>>>>>>> sent over the wire and how rsyslog has parsed it.
>> >>>>>>>>>>>
>> >>>>>>>>>>> David Lang
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
>> >>>>>>>>>>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
>> >>>>>>>>>>>> To: John Chivian <jchivian@chivian.com>
>> >>>>>>>>>>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
>> >>>>>>>>>>>> rsyslog-users <rsyslog@lists.adiscon.com>
>> >>>>>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Thanks, John, let me try to clarify what I mean.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Normally when I forward from a remote server to the central
>> log
>> >>>>>>>>>> server, I
>> >>>>>>>>>>>> can include a tag that can then be used to determine the file
>> >>>>>> name I
>> >>>>>>>>>> want
>> >>>>>>>>>>>> on the central server. Since I have no real way to include
>> this
>> >>>>>> tag
>> >>>>>>>>>> from
>> >>>>>>>>>>>> the appliance, this is not an option.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> I'm looking for a way of inspecting the incoming packets to
>> >>>>>>>> determining
>> >>>>>>>>>>> the
>> >>>>>>>>>>>> source logfile name (which is included in the payload) and
>> use
>> >>>>>> that
>> >>>>>>>>>>>> filename on the target central server. Since there are
>> multiple
>> >>>>>> logs
>> >>>>>>>>>>> being
>> >>>>>>>>>>>> sent (access, audit, monitor, etc.), I'd like to segregate
>> these
>> >>>>>> into
>> >>>>>>>>>>> their
>> >>>>>>>>>>>> own files. I'm already using a template with the host
>> >> information
>> >>>>>> to
>> >>>>>>>>>>>> dynamically create the file names. I just don't know how I
>> can
>> >> go
>> >>>>>>>>>> beyond
>> >>>>>>>>>>>> this to also include the source logname.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Here's the template I'm using. It works for all other hosts
>> >> where
>> >>>>>> I
>> >>>>>>>> can
>> >>>>>>>>>>>> configure the tag but I get garbage names from the
>> appliance. I
>> >>>>>> had
>> >>>>>>>>>> hoped
>> >>>>>>>>>>>> that the appliance included some standard syslog tags but it
>> >>>>>> doesn't
>> >>>>>>>>>> seem
>> >>>>>>>>>>>> so.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> template(name="DynRemoteLogFile" type="string"
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> *Scott Slattery*
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> *Sr. Systems & Cloud Architect*
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> motorolasolutions.com
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> *O: 602.529.8226*
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <
>> >>>>>> jchivian@chivian.com>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> Your use of the term “file name” is confusing. When senders
>> >>>>>> deliver
>> >>>>>>>>>> to
>> >>>>>>>>>>>>> rsyslog over the network there is no exchange of files or
>> >>>>>> filenames,
>> >>>>>>>>>>> only
>> >>>>>>>>>>>>> packets of information. Those packets are expected to be
>> in a
>> >>>>>> format
>> >>>>>>>>>>> that
>> >>>>>>>>>>>>> syslog understands such that useful information (header
>> >> elements
>> >>>>>> and
>> >>>>>>>>>>>>> message body) may be parsed from them. If you as the
>> rsyslog
>> >>>>>> admin
>> >>>>>>>>>>> choose
>> >>>>>>>>>>>>> to use some of that header information to compose filenames
>> for
>> >>>>>>>> output
>> >>>>>>>>>>>>> files, then yes you are sort of at the mercy of the senders
>> >>>>>> content
>> >>>>>>>>>>>>> (especially if the sender doesn’t follow the syslog rules).
>> >>>>>> However,
>> >>>>>>>>>>> there
>> >>>>>>>>>>>>> are functions in the advanced syntax that can be used to
>> >> perform
>> >>>>>> the
>> >>>>>>>>>>> type
>> >>>>>>>>>>>>> of character replacements you’re talking about.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> It is common practice to use the syslog header/rsyslog
>> property
>> >>>>>>>>>> element
>> >>>>>>>>>>>>> called “hostname” for just such purposes. Is this what
>> you’re
>> >>>>>>>> talking
>> >>>>>>>>>>>>> about? You’d have to provide your configuration for real
>> >>>>>> analysis,
>> >>>>>>>> at
>> >>>>>>>>>>>>> least the part you perceive to be responsible for the
>> problem.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Regards,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
>> >>>>>>>>>>>>> rsyslog@lists.adiscon.com> wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> I have a configured central log collector using rsyslog. A
>> few
>> >>>>>> of
>> >>>>>>>>>> the
>> >>>>>>>>>>>>>> devices forwarding their logs are appliances that have no
>> >>>>>>>>>>> configuration
>> >>>>>>>>>>>>>> options other than the IP forwarding address and protocol.
>> I
>> >>>>>> cannot
>> >>>>>>>>>>>>> control
>> >>>>>>>>>>>>>> what file names are being sent.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Unfortunately, they are sending unintelligible file names
>> with
>> >>>>>>>>>>> characters
>> >>>>>>>>>>>>>> that normally would be escaped. Is there any way I can
>> control
>> >>>>>> or
>> >>>>>>>>>>> alter
>> >>>>>>>>>>>>> the
>> >>>>>>>>>>>>>> incoming file name to normalize it to avoid these odd
>> >>>>>> characters?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> For example, could I establish a character map that maps
>> the
>> >>>>>>>>>> unallowed
>> >>>>>>>>>>>>>> character to something acceptable?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> thanks,
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> *Scott Slattery*
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> *Sr. Systems & Cloud Architect*
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> motorolasolutions.com
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> *O: 602.529.822*
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> --
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> *For more information on how and why we collect your
>> personal
>> >>>>>>>>>>>>>> information, please visit our Privacy Policy
>> >>>>>>>>>>>>>> <
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>> >>>>>>>>>>>>>> .*
>> >>>>>>>>>>>>>> _______________________________________________
>> >>>>>>>>>>>>>> rsyslog mailing list
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
>> >>>>>>>>>>>>>> What's up with rsyslog? Follow
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
>> >>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
>> ARCHIVED
>> >> by
>> >>>>>> a
>> >>>>>>>>>>> myriad
>> >>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST
>> >>>>>> if
>> >>>>>>>> you
>> >>>>>>>>>>>>> DON'T LIKE THAT.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> --
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> *For more information on how and why we collect your personal
>> >>>>>>>>>>>> information, please visit our Privacy Policy
>> >>>>>>>>>>>> <
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>> >>>>>>>>>>>> .*
>> >>>>>>>>>>>> _______________________________________________
>> >>>>>>>>>>>> rsyslog mailing list
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
>> >>>>>>>>>>>> What's up with rsyslog? Follow
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
>> >>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>> by
>> >> a
>> >>>>>>>>>> myriad
>> >>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST
>> >> if
>> >>>>>> you
>> >>>>>>>>>>> DON'T LIKE THAT.
>> >>>>>>>>>>
>> >>>>>>>>>> --
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> *For more information on how and why we collect your personal
>> >>>>>>>>>> information, please visit our Privacy Policy
>> >>>>>>>>>> <
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>> >>>>>>>>>>> .*
>> >>>>>>>>>> _______________________________________________
>> >>>>>>>>>> rsyslog mailing list
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
>> >>>>>>>>>> What's up with rsyslog? Follow
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
>> >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>> by a
>> >>>>>> myriad
>> >>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST if
>> >>>>>> you
>> >>>>>>>>>> DON'T LIKE THAT.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>
>> >>>>>>> --
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> *For more information on how and why we collect your personal
>> >>>>>>> information, please visit our Privacy Policy
>> >>>>>>> <
>> >>>>>>
>> >>>>
>> >>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>> >>>>>>> .*
>> >>>>>>> _______________________________________________
>> >>>>>>> rsyslog mailing list
>> >>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=7hc9b2gSfvsfQzXvsRVXqGeP4Rz_tJtrykpjqkGBIdk&e=
>> >>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=bDybhvts9KqI-WEfMHo7O40ulQK5M6Tg4tnKNRRIhE8&e=
>> >>>>>>> What's up with rsyslog? Follow
>> >>>>>>
>> >>>>
>> >>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=zPhpcUPlK_CWLiU75_WwjyQGz5mFTiXc1yyBIfppkrI&e=
>> >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> >>>> myriad
>> >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you
>> >>>>>> DON'T LIKE THAT.
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>
>> >>>
>> >
>> >
>
>

--


*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Altering forwarded logfile names [ In reply to ]
I don't think this is the problem, but 8.24 is 4+ years old now (releaseed Jan
2017, although the version maintained by RedHat and Amazon has some bugfixes
backported), can you try with a current version (8.2102 or 8.2012) and see if
you still have the problem. I don't remember of hearing about a similar problem
several years ago, but since we don't know what patches that version has
compared to the version the commmunity released, it's hard to say. There are a
LOT of bugfixes and new features introduced over that time.

thanks for the config, I don't see anything odd in it.

David Lang

On Thu, 25 Mar 2021, Scott Slattery
wrote:

> Date: Thu, 25 Mar 2021 15:24:23 -0700
> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> To: David Lang <david@lang.hm>
> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
> rsyslog-users <rsyslog@lists.adiscon.com>, mariusz.kruk@safecomp.com
> Subject: Re: [rsyslog] Altering forwarded logfile names
>
> *Rsyslogd version:*
> rsyslogd 8.24.0-52.amzn2, compiled with:
> PLATFORM: x86_64-koji-linux-gnu
> PLATFORM (lsb_release -d):
> FEATURE_REGEXP: Yes
> GSSAPI Kerberos 5 support: Yes
> FEATURE_DEBUG (debug build, slow code): No
> 32bit Atomic operations supported: Yes
> 64bit Atomic operations supported: Yes
> memory allocator: system default
> Runtime Instrumentation (slow code): No
> uuid support: Yes
> Number of Bits in RainerScript integers: 64
>
> Here is my config:
> $MaxMessageSize 64k
> $EscapeControlCharactersOnReceive on
>
> module(load="imuxsock")
> module(load="imjournal" StateFile="imjournal.state")
> module(load="imklog")
> module(load="immark")
> module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat"
> DirCreateMode="0705" FileCreateMode="0704")
> module(load="impstats" interval="600" log.syslog="off" severity="7"
> log.file="/var/log/impstats.log")
>
> $DynaFileCacheSize 100
> global(workDirectory="/var/lib/rsyslog" localHostname="cloudlogmaster01")
>
> $MainMsgQueueSize 200000
> $MainMsgQueueHighWaterMark 120000
> $MainMsgQueueDiscardMark 160000
> $MainMsgQueueWorkerThreads 100
> $RulesetCreateMainQueue on
>
> $imjournalRatelimitInterval 0
> $imjournalRatelimitBurst 0
> #### RULES ####
>
> template(name="debug" type="string" string="Debug line with all
> properties:\nFROMHOST: '%FROMHOST%'\n, FROMHOST-IP: '%fromhost-ip%'\n,
> HOSTNAME: '%HOSTNAME%'\n, PRI: %PRI%,\nSYSLOGTAG '%syslogtag%'\n,
> PROGRAMNAME: '%programname%'\n, APP-NAME: '%app-name%'\n, PROCID:
> '%PROCID%'\n, MSGID: '%MSGID%'\n, TIMESTAMP: '%TIMESTAMP%'\n,
> STRUCTURED-DATA: '%STRUCTURED-DATA%'\n, msg: '%msg%'\nRAWMSG:
> '%rawmsg%',\n\nRAWMSG-AFTER-PRI: '%rawmsg-after-pri%'\nJSONMESG:
> '%jsonmesg%'\n")
>
> template(name="DynRemoteLogFile" type="string"
> string="/splunklog/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
>
> template(name="MalformedMsgFormatter" type="string" string="%timegenerated%
> %fromhost% %rawmsg:::drop-last-lf%\n")
> template(name="DynRemoteMalformed" type="string"
> string="/splunklog/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-oag-audit.log")
>
> template(name="DynRemoteLogFile" type="list") {
> constant(value="/splunklog/remote/")
> property(name="FROMHOST")
> constant(value="-")
> property(name="FROMHOST-IP")
> constant(value="/")
> property(name="$YEAR")
> constant(value="-")
> property(name="$MONTH")
> constant(value="-")
> property(name="$DAY")
> constant(value="-")
> property(name="syslogtag" position.from="1" position.to="32")
> constant(value="-")
> property(name="app-name" controlcharacters="drop")
> constant(value=".log")
> }
>
> ruleset(name="RemoteServer"){
> if ($fromhost-ip startswith '10.41.102') then {
> action(type="omfile" dynafile="DynRemoteMalformed"
> template="MalformedMsgFormatter")
> stop
> }
>
> action(type="omfile" dynaFile="DynRemoteLogFile")
> }
>
>
> # module imtcp
> module(load="imtcp" KeepAlive="on")
> input(type="imtcp" port="10514" ruleset="RemoteServer")
>
>
> #kern.* /dev/console
> *.info;mail.none;authpriv.none;cron.none /var/log/messages
> authpriv.* /var/log/secure
> mail.* /var/log/maillog
> cron.* /var/log/cron
> *.emerg :omusrmsg:*
> uucp,news.crit /var/log/spooler
> local7.* /var/log/boot.log
>
> *Scott Slattery*
>
> *Sr. Systems & Cloud Architect*
>
> *Cloud, Compute, Information & Architecture Team*
>
> motorolasolutions.com
>
> *O: 602.529.8226*
>
> *E*: Scott.Slattery@MotorolaSolutions.com
>
>
>
>
> On Thu, Mar 25, 2021 at 3:20 PM David Lang <david@lang.hm> wrote:
>
>> what version of rsyslog are you running. can you post your full config?
>>
>> if you are receiving via TCP and it's not splitting the logs based on
>> newlines,
>> something very odd is happening.
>>
>> David Lang
>>
>> On Thu, 25 Mar 2021, Scott Slattery wrote:
>>
>>> Date: Thu, 25 Mar 2021 15:07:22 -0700
>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>> To: David Lang <david@lang.hm>
>>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
>>> rsyslog-users <rsyslog@lists.adiscon.com>, mariusz.kruk@safecomp.com
>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>
>>> So I'm looking at the pcap and on the first log record the header
>>> information looks fine and the record terminated by 0x0A line feed.
>>>
>>> When I look at the record containing this record written to the rsyslog
>>> file, the record length is 65586 (64k) long which is what I think you
>> were
>>> referring to earlier. Why would rsyslog write such a long record? I
>>> initially thought my data may be getting truncated so I updated
>>> '$MaxMessageSize 64k'. Any suggestion how I fix this behavior.
>>>
>>> I can confirm that each log record in the pcap is terminated by 0x0A and
>>> looks in tact so must be something I need to do on rsyslog to fix this.
>>>
>>> *Scott Slattery*
>>>
>>> *Sr. Systems & Cloud Architect*
>>>
>>> *Cloud, Compute, Information & Architecture Team*
>>>
>>> motorolasolutions.com
>>>
>>> *O: 602.529.8226*
>>>
>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>
>>>
>>>
>>>
>>> On Thu, Mar 25, 2021 at 1:15 PM David Lang <david@lang.hm> wrote:
>>>
>>>> you may want to capture with -X so that it decodes it into hex and you
>> can
>>>> see
>>>> newlines vs linefeeds
>>>>
>>>> David Lang
>>>>
>>>> On Thu, 25 Mar 2021, Scott Slattery wrote:
>>>>
>>>>> Date: Thu, 25 Mar 2021 12:47:43 -0700
>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>>>> To: David Lang <david@lang.hm>
>>>>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
>>>>> rsyslog-users <rsyslog@lists.adiscon.com>,
>> mariusz.kruk@safecomp.com
>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>>>
>>>>> Thanks David, I just grabbed my first capture and am looking at it. I'm
>>>>> using TCP vs UDP but your comment makes sense. I'll certainly share
>> with
>>>>> you my observations once I have some data to refer to. This has been a
>>>>> learning experience. It's also the first time I've ever seen such an
>>>> issue
>>>>> with rsyslog so it's a bit of a puzzle.
>>>>>
>>>>> *Scott Slattery*
>>>>>
>>>>> *Sr. Systems & Cloud Architect*
>>>>>
>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>
>>>>> motorolasolutions.com
>>>>>
>>>>> *O: 602.529.8226*
>>>>>
>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Mar 25, 2021 at 12:32 PM David Lang <david@lang.hm> wrote:
>>>>>
>>>>>> the rawmsg field in the debugformat output shows exactly what rsyslog
>> is
>>>>>> seeing.
>>>>>>
>>>>>> the reason I asked you to check multiple entries is that if rsyslog
>> does
>>>>>> not see
>>>>>> the separator (due to either multiple messages in one UDP packet, or
>>>>>> missing
>>>>>> newlines in a TCP stream) it will combine what are intended as
>> multiple
>>>>>> messages
>>>>>> together into one message as far as rsyslog is concerned, and only
>> break
>>>>>> it into
>>>>>> multiple messages when it hits the configured maxmessagesize size.
>> That
>>>>>> breaks
>>>>>> things at arbitrary points in the message, so as you look at multiple
>>>>>> messages,
>>>>>> they will not all start the same way.
>>>>>>
>>>>>> a tcpdump -s0 -A -n (with the appropriate port/ip filter) will also
>> show
>>>>>> exactly
>>>>>> what's happening over the wire.
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>> On Thu, 25 Mar 2021, Scott Slattery wrote:
>>>>>>
>>>>>>> Date: Thu, 25 Mar 2021 12:23:03 -0700
>>>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>>>>>> To: Rainer Gerhards <rgerhards@hq.adiscon.com>
>>>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
>>>> david@lang.hm
>>>>>>> ,
>>>>>>> mariusz.kruk@safecomp.com
>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>>>>>
>>>>>>> Hey Guys, before I consume more of your time, I'm capturing the
>> packets
>>>>>>> from the source device to determine if any of the payload is lost on
>>>> its
>>>>>>> way to the rsyslog collector. After I analyze further, I'll share my
>>>>>>> observations. I'm hoping nothing is getting dropped over the wire.
>>>> Thanks
>>>>>>> for your patience and help so far. I expect to have additional
>> details
>>>>>>> shortly.
>>>>>>>
>>>>>>> *Scott Slattery*
>>>>>>>
>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>
>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>
>>>>>>> motorolasolutions.com
>>>>>>>
>>>>>>> *O: 602.529.8226*
>>>>>>>
>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Mar 25, 2021 at 12:51 AM Rainer Gerhards <
>>>>>> rgerhards@hq.adiscon.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Sorry if I didn't catch it, but did you post the configuration? If
>>>>>>>> not, it would probably be useful.
>>>>>>>>
>>>>>>>> Looking at the message properties, this looks like the message was
>>>>>>>> received via UDP but had a LF inside it which was intended as a
>> frame
>>>>>>>> separator. If so, that is another bug in the vendor's
>> implementation.
>>>>>>>> Framing is used for TCP, but not UDP. For UDP it is strictly one
>>>>>>>> message per package.
>>>>>>>>
>>>>>>>> HTH,
>>>>>>>> Rainer
>>>>>>>>
>>>>>>>> El mié, 24 mar 2021 a las 22:23, Scott Slattery via rsyslog
>>>>>>>> (<rsyslog@lists.adiscon.com>) escribió:
>>>>>>>>>
>>>>>>>>> From what I can discern since there are lots of transactions, there
>>>>>> seem
>>>>>>>> to
>>>>>>>>> be multiple payload formats. For all that I have so far analyzed,
>> the
>>>>>>>> date
>>>>>>>>> truncation issue is the same regardless of the payload received.
>> Like
>>>>>>>> you,
>>>>>>>>> I've never used mmnormalize but I will look at it as an option.
>> I've
>>>>>>>>> reached out to the product support team to better understand if
>> there
>>>>>> is
>>>>>>>> a
>>>>>>>>> appliance-side misconfiguration and something that can be corrected
>>>>>> there
>>>>>>>>> or if I need to make adjustments on the rsyslog side to clean up
>> the
>>>>>>>>> records.
>>>>>>>>>
>>>>>>>>> What I meant is that there are multiple message formats (payload
>>>> data)
>>>>>>>> but
>>>>>>>>> there seems to be a consistent truncation of the date field just as
>>>> you
>>>>>>>> saw
>>>>>>>>> in the app-name tag field.
>>>>>>>>>
>>>>>>>>> I appreciate your ongoing feedback and help. This issue is new to
>> me
>>>>>> and
>>>>>>>>> will need a creative solution if the vendor has no answers.
>>>>>>>>>
>>>>>>>>> Many thanks,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *Scott Slattery*
>>>>>>>>>
>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>>
>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>>
>>>>>>>>> motorolasolutions.com
>>>>>>>>>
>>>>>>>>> *O: 602.529.8226*
>>>>>>>>>
>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Mar 24, 2021 at 2:14 PM David Lang <david@lang.hm> wrote:
>>>>>>>>>
>>>>>>>>>> when you say 'each is incorrect but the same format'
>>>>>>>>>>
>>>>>>>>>> does that mean that every log has the year missing? or that every
>>>> log
>>>>>>>> is
>>>>>>>>>> combining the logs together?
>>>>>>>>>>
>>>>>>>>>> I'll note that it's possible to define a custom parser using the
>>>>>>>>>> mmnormalize
>>>>>>>>>> library and add it to the parsing stack. I helped define the need
>>>> for
>>>>>>>> such
>>>>>>>>>> a
>>>>>>>>>> capability, but haven't used it yet.
>>>>>>>>>>
>>>>>>>>>> David Lang
>>>>>>>>>>
>>>>>>>>>> On Wed, 24 Mar 2021, Scott Slattery wrote:
>>>>>>>>>>
>>>>>>>>>>> Date: Wed, 24 Mar 2021 08:58:07 -0700
>>>>>>>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>>>>>>>>>> To: mariusz.kruk@safecomp.com
>>>>>>>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
>>>>>>>> david@lang.hm
>>>>>>>>>>>
>>>>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>>>>>>>>>
>>>>>>>>>>> Yes, all the logs are coming in this way. There are three
>> separate
>>>>>>>> logs
>>>>>>>>>>> being sent and each is incorrect in its own right but the same
>>>> format
>>>>>>>>>>> behavior is consistent across each of the three logs. I will look
>>>>>>>> further
>>>>>>>>>>> into the suggestions you have provided and have already reached
>> out
>>>>>>>> to
>>>>>>>>>> the
>>>>>>>>>>> appliance support team and will raise the nonconformance with the
>>>>>>>> RFC.
>>>>>>>>>>>
>>>>>>>>>>> This does provide me some valuable guidance and I thank each of
>> you
>>>>>>>> for
>>>>>>>>>>> your feedback.
>>>>>>>>>>>
>>>>>>>>>>> *Scott Slattery*
>>>>>>>>>>>
>>>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>>>>
>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>>>>
>>>>>>>>>>> motorolasolutions.com
>>>>>>>>>>>
>>>>>>>>>>> *O: 602.529.8226*
>>>>>>>>>>>
>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com>
>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> There's something strange with this log entry. Notice that it
>>>> looks
>>>>>>>> as
>>>>>>>>>> if
>>>>>>>>>>>> it were more than one log entries joined with LF character
>>>> (escaped
>>>>>>>> on
>>>>>>>>>>>> input as #012). And all "further" entries start with a - still
>>>>>>>>>>>> nonconformant to the RFC but "fuller" - timestamp containing
>>>>>>>> year.It's
>>>>>>>>>> the
>>>>>>>>>>>> leading event that seems to be trimmed in the front somehow. Do
>>>> all
>>>>>>>>>> events
>>>>>>>>>>>> arrive like this?Wys?ano z telefonu Samsung<div>
>>>>>>>>>>>> </div><div>
>>>>>>>>>>>> </div><!-- originalMessage --><div>-------- Original message
>>>>>>>>>>>> --------</div><div>From: Scott Slattery via rsyslog <
>>>>>>>>>>>> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021 01:52
>>>>>>>>>>>> (GMT+01:00) </div><div>To: David Lang <david@lang.hm>
>>>>>>>> </div><div>Cc:
>>>>>>>>>>>> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott
>>>>>>>> Slattery
>>>>>>>>>> via
>>>>>>>>>>>> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re:
>>>>>>>> [rsyslog]
>>>>>>>>>>>> Altering forwarded logfile names </div><div>
>>>>>>>>>>>> </div>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi David, fortunately I had already done this. I'm including an
>>>>>>>> actual
>>>>>>>>>> log
>>>>>>>>>>>> entry but have anonymized the data to keep the actual user and
>>>> email
>>>>>>>>>>>> address confidential:
>>>>>>>>>>>>
>>>>>>>>>>>> Debug line with all properties:
>>>>>>>>>>>> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
>>>>>>>>>>>> , fromhost-ip: '10.41.102.143'
>>>>>>>>>>>> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
>>>>>>>>>>>> , PRI: 13,
>>>>>>>>>>>> syslogtag '03-23T16:'
>>>>>>>>>>>> , programname: '03-23T16'
>>>>>>>>>>>> , APP-NAME: '03-23T16'
>>>>>>>>>>>> , PROCID: '-'
>>>>>>>>>>>> , MSGID: '-'
>>>>>>>>>>>> , TIMNESTAMP: 'Mar 23 21:47:13'
>>>>>>>>>>>> , STRUCTURED-DATA: '-'
>>>>>>>>>>>> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
>>>>>>>>>> ACCESS_GATEWAY
>>>>>>>>>>>> ACCESS AUTHZ SESSION INFO USER_SESSION
>>>>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>> SUBJECT="
>>>>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>>>>>> apigniodashboard-uat.mycompany.com"
>>>>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>>>>>>>>> REMOTE_IP="10.44.65.38"
>>>>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>>>>>> Safari/537.36"]
>>>>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>>> POLICY
>>>>>>>>>> INFO
>>>>>>>>>>>> USER_AUTHZ
>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
>>>> RESOURCE="/_dash-dependencies"
>>>>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
>>>>>>>>>> APP="Ignio
>>>>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A
>> -
>>>>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
>>>> oag_username=
>>>>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>>>>>>>>>> apigniodashboard-uat.mycompany.com
>>>>>>>>>>>> UserName=Joe.User@mycompantions.com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
>> Mac
>>>>>>>> OS X
>>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>>> Chrome/88.0.4324.192
>>>>>>>>>>>> Safari/537.36 creationTime=1616536016811
>>>> maxInactiveInterval=3600000
>>>>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
>> Intel
>>>>>>>> Mac
>>>>>>>>>> OS X
>>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>>> Chrome/88.0.4324.192
>>>>>>>>>>>> Safari/537.36"] allow access to
>>>>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>>>>>>> SESSION
>>>>>>>>>> INFO
>>>>>>>>>>>> USER_SESSION
>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>>>>>> SUBJECT="Joe.User@mycompan'
>>>>>>>>>>>> *escaped msg*: '47:20.708-05:00
>> AUSE1OAGATST02.aws.mycompany.com
>>>>>>>>>>>> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
>>>>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>> SUBJECT="
>>>>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>>>>>> apigniodashboard-uat.mycompany.com"
>>>>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>>>>>>>>> REMOTE_IP="10.44.65.38"
>>>>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>>>>>> Safari/537.36"]
>>>>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>>> POLICY
>>>>>>>>>> INFO
>>>>>>>>>>>> USER_AUTHZ
>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
>>>> RESOURCE="/_dash-dependencies"
>>>>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
>>>>>>>>>> APP="Ignio
>>>>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A
>> -
>>>>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
>>>> oag_username=
>>>>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>>>>>>>>>> apigniodashboard-uat.mycompany.com
>>>>>>>>>>>> UserName=Joe.User@mycompantions.com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
>> Mac
>>>>>>>> OS X
>>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>>> Chrome/88.0.4324.192
>>>>>>>>>>>> Safari/537.36 creationTime=1616536016811
>>>> maxInactiveInterval=3600000
>>>>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
>> Intel
>>>>>>>> Mac
>>>>>>>>>> OS X
>>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>>> Chrome/88.0.4324.192
>>>>>>>>>>>> Safari/537.36"] allow access to
>>>>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>>>>>>> SESSION
>>>>>>>>>> INFO
>>>>>>>>>>>> USER_SESSION
>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>>>>>> SUBJECT="Joe.User@mycompan'
>>>>>>>>>>>> *rawmsg*: '03-23T16:47:20.708-05:00
>>>>>>>> AUSE1OAGATST02.aws.mycompany.com
>>>>>>>>>>>> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
>>>>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>> SUBJECT="
>>>>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>>>>>> apigniodashboard-uat.mycompany.com"
>>>>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>>>>>>>>> REMOTE_IP="10.44.65.38"
>>>>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>>>>>> Safari/537.36"]
>>>>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>>> POLICY
>>>>>>>>>> INFO
>>>>>>>>>>>> USER_AUTHZ
>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
>>>> RESOURCE="/_dash-dependencies"
>>>>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
>>>>>>>>>> APP="Ignio
>>>>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A
>> -
>>>>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
>>>> oag_username=
>>>>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>>>>>>>>>> apigniodashboard-uat.mycompany.com
>>>>>>>>>>>> UserName=Joe.User@mycompantions.com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
>> Mac
>>>>>>>> OS X
>>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>>> Chrome/88.0.4324.192
>>>>>>>>>>>> Safari/537.36 creationTime=1616536016811
>>>> maxInactiveInterval=3600000
>>>>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
>> Intel
>>>>>>>> Mac
>>>>>>>>>> OS X
>>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>>> Chrome/88.0.4324.192
>>>>>>>>>>>> Safari/537.36"] allow access to
>>>>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>>>>>>> SESSION
>>>>>>>>>> INFO
>>>>>>>>>>>> USER_SESSION
>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>>>>>> SUBJECT="Joe.User@mycompan'
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> By inspecting the rawmsg, I can see that field four
>>>>>>>> (space-delimited)
>>>>>>>>>>>> indicates this is the ACCESS log. So if I were able to extract
>> the
>>>>>>>> log
>>>>>>>>>>>> identifier from the msg, I could then write all access logs to
>> the
>>>>>>>> same
>>>>>>>>>>>> daily file. There are other formats as well from the same device
>>>>>>>> but the
>>>>>>>>>>>> idea is the same.
>>>>>>>>>>>>
>>>>>>>>>>>> *Scott Slattery*
>>>>>>>>>>>>
>>>>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>>>>>
>>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>>>>>
>>>>>>>>>>>> motorolasolutions.com
>>>>>>>>>>>>
>>>>>>>>>>>> *O: 602.529.8226*
>>>>>>>>>>>>
>>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm>
>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> the source logfile name is not included in the payload by the
>>>>>>>> syslog
>>>>>>>>>>>> spec.
>>>>>>>>>>>>> It
>>>>>>>>>>>>> may be in the case of your appliance, but we would need to see
>> a
>>>>>>>> sample
>>>>>>>>>>>>> log to
>>>>>>>>>>>>> understand ho to parse it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> based on your template, you are using app-name, which may be
>>>> listed
>>>>>>>>>>>>> separtely if
>>>>>>>>>>>>> it's a RFC5424 format log, or may be part of the syslog tag if
>>>>>>>> it's a
>>>>>>>>>>>>> RFC3164
>>>>>>>>>>>>> format log over the wire (neither format has a way to specify a
>>>>>>>> source
>>>>>>>>>>>> log
>>>>>>>>>>>>> file
>>>>>>>>>>>>> by default)
>>>>>>>>>>>>>
>>>>>>>>>>>>> you can look at
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
>>>>>>>>>>>>> and see the *-cc
>>>>>>>>>>>>> options that you could apply to the app-name to eliminate
>> control
>>>>>>>>>>>>> characters.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Again, we really need to see the original log message to
>>>> understand
>>>>>>>>>>>> what's
>>>>>>>>>>>>> what.
>>>>>>>>>>>>> Please log it with the templateRSYSLOG_DebugFormat so we can
>> see
>>>>>>>>>> exactly
>>>>>>>>>>>>> what is
>>>>>>>>>>>>> sent over the wire and how rsyslog has parsed it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> David Lang
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
>>>>>>>>>>>>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
>>>>>>>>>>>>>> To: John Chivian <jchivian@chivian.com>
>>>>>>>>>>>>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
>>>>>>>>>>>>>> rsyslog-users <rsyslog@lists.adiscon.com>
>>>>>>>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks, John, let me try to clarify what I mean.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Normally when I forward from a remote server to the central
>> log
>>>>>>>>>>>> server, I
>>>>>>>>>>>>>> can include a tag that can then be used to determine the file
>>>>>>>> name I
>>>>>>>>>>>> want
>>>>>>>>>>>>>> on the central server. Since I have no real way to include
>> this
>>>>>>>> tag
>>>>>>>>>>>> from
>>>>>>>>>>>>>> the appliance, this is not an option.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm looking for a way of inspecting the incoming packets to
>>>>>>>>>> determining
>>>>>>>>>>>>> the
>>>>>>>>>>>>>> source logfile name (which is included in the payload) and use
>>>>>>>> that
>>>>>>>>>>>>>> filename on the target central server. Since there are
>> multiple
>>>>>>>> logs
>>>>>>>>>>>>> being
>>>>>>>>>>>>>> sent (access, audit, monitor, etc.), I'd like to segregate
>> these
>>>>>>>> into
>>>>>>>>>>>>> their
>>>>>>>>>>>>>> own files. I'm already using a template with the host
>>>> information
>>>>>>>> to
>>>>>>>>>>>>>> dynamically create the file names. I just don't know how I can
>>>> go
>>>>>>>>>>>> beyond
>>>>>>>>>>>>>> this to also include the source logname.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here's the template I'm using. It works for all other hosts
>>>> where
>>>>>>>> I
>>>>>>>>>> can
>>>>>>>>>>>>>> configure the tag but I get garbage names from the appliance.
>> I
>>>>>>>> had
>>>>>>>>>>>> hoped
>>>>>>>>>>>>>> that the appliance included some standard syslog tags but it
>>>>>>>> doesn't
>>>>>>>>>>>> seem
>>>>>>>>>>>>>> so.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> template(name="DynRemoteLogFile" type="string"
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Scott Slattery*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> motorolasolutions.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *O: 602.529.8226*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <
>>>>>>>> jchivian@chivian.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Your use of the term “file name” is confusing. When senders
>>>>>>>> deliver
>>>>>>>>>>>> to
>>>>>>>>>>>>>>> rsyslog over the network there is no exchange of files or
>>>>>>>> filenames,
>>>>>>>>>>>>> only
>>>>>>>>>>>>>>> packets of information. Those packets are expected to be in
>> a
>>>>>>>> format
>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> syslog understands such that useful information (header
>>>> elements
>>>>>>>> and
>>>>>>>>>>>>>>> message body) may be parsed from them. If you as the rsyslog
>>>>>>>> admin
>>>>>>>>>>>>> choose
>>>>>>>>>>>>>>> to use some of that header information to compose filenames
>> for
>>>>>>>>>> output
>>>>>>>>>>>>>>> files, then yes you are sort of at the mercy of the senders
>>>>>>>> content
>>>>>>>>>>>>>>> (especially if the sender doesn’t follow the syslog rules).
>>>>>>>> However,
>>>>>>>>>>>>> there
>>>>>>>>>>>>>>> are functions in the advanced syntax that can be used to
>>>> perform
>>>>>>>> the
>>>>>>>>>>>>> type
>>>>>>>>>>>>>>> of character replacements you’re talking about.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It is common practice to use the syslog header/rsyslog
>> property
>>>>>>>>>>>> element
>>>>>>>>>>>>>>> called “hostname” for just such purposes. Is this what
>> you’re
>>>>>>>>>> talking
>>>>>>>>>>>>>>> about? You’d have to provide your configuration for real
>>>>>>>> analysis,
>>>>>>>>>> at
>>>>>>>>>>>>>>> least the part you perceive to be responsible for the
>> problem.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
>>>>>>>>>>>>>>> rsyslog@lists.adiscon.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have a configured central log collector using rsyslog. A
>> few
>>>>>>>> of
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> devices forwarding their logs are appliances that have no
>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>>> options other than the IP forwarding address and protocol. I
>>>>>>>> cannot
>>>>>>>>>>>>>>> control
>>>>>>>>>>>>>>>> what file names are being sent.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Unfortunately, they are sending unintelligible file names
>> with
>>>>>>>>>>>>> characters
>>>>>>>>>>>>>>>> that normally would be escaped. Is there any way I can
>> control
>>>>>>>> or
>>>>>>>>>>>>> alter
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> incoming file name to normalize it to avoid these odd
>>>>>>>> characters?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> For example, could I establish a character map that maps the
>>>>>>>>>>>> unallowed
>>>>>>>>>>>>>>>> character to something acceptable?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *Scott Slattery*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> motorolasolutions.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *O: 602.529.822*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *For more information on how and why we collect your
>> personal
>>>>>>>>>>>>>>>> information, please visit our Privacy Policy
>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>>>>>>>>>>>> .*
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
>>>>>>>>>>>>>>>> What's up with rsyslog? Follow
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
>>>>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>> by
>>>>>>>> a
>>>>>>>>>>>>> myriad
>>>>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST
>>>>>>>> if
>>>>>>>>>> you
>>>>>>>>>>>>>>> DON'T LIKE THAT.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *For more information on how and why we collect your personal
>>>>>>>>>>>>>> information, please visit our Privacy Policy
>>>>>>>>>>>>>> <
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>>>>>>>>>> .*
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
>>>>>>>>>>>>>> What's up with rsyslog? Follow
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
>>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>> by
>>>> a
>>>>>>>>>>>> myriad
>>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>>> if
>>>>>>>> you
>>>>>>>>>>>>> DON'T LIKE THAT.
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *For more information on how and why we collect your personal
>>>>>>>>>>>> information, please visit our Privacy Policy
>>>>>>>>>>>> <
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>>>>>>>>> .*
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
>>>>>>>>>>>> What's up with rsyslog? Follow
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>> a
>>>>>>>> myriad
>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if
>>>>>>>> you
>>>>>>>>>>>> DON'T LIKE THAT.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *For more information on how and why we collect your personal
>>>>>>>>> information, please visit our Privacy Policy
>>>>>>>>> <
>>>>>>>>
>>>>>>
>>>>
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>>>>> .*
>>>>>>>>> _______________________________________________
>>>>>>>>> rsyslog mailing list
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=7hc9b2gSfvsfQzXvsRVXqGeP4Rz_tJtrykpjqkGBIdk&e=
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=bDybhvts9KqI-WEfMHo7O40ulQK5M6Tg4tnKNRRIhE8&e=
>>>>>>>>> What's up with rsyslog? Follow
>>>>>>>>
>>>>>>
>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=zPhpcUPlK_CWLiU75_WwjyQGz5mFTiXc1yyBIfppkrI&e=
>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>> myriad
>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you
>>>>>>>> DON'T LIKE THAT.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

1 2  View All