There's something strange with this log entry. Notice that it looks as
if it were more than one log entries joined with LF character (escaped
on input as #012). And all "further" entries start with a - still
nonconformant to the RFC but "fuller" - timestamp containing year.
It's the leading event that seems to be trimmed in the front somehow. Do
all events arrive like this?
It looks as if the input wasn't properly splitting the stream into
single entries. Did you touch the frame delimiter setting on input?
Wys?ano z telefonu Samsung
-------- Original message --------
From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
Date: 24/03/2021 01:52 (GMT+01:00)
To: David Lang <david@lang.hm>
Cc: Scott Slattery <scott.slattery@motorolasolutions.com>, Scott
Slattery via rsyslog <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Altering forwarded logfile names
Hi David, fortunately I had already done this. I'm including an actual log
entry but have anonymized the data to keep the actual user and email
address confidential:
Debug line with all properties:
FROMHOST: 'ause1oagatst02.aws.mycompany.com'
, fromhost-ip: '10.41.102.143'
, HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
, PRI: 13,
syslogtag '03-23T16:'
, programname: '03-23T16'
, APP-NAME: '03-23T16'
, PROCID: '-'
, MSGID: '-'
, TIMNESTAMP: 'Mar 23 21:47:13'
, STRUCTURED-DATA: '-'
, *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY
ACCESS AUTHZ SESSION INFO USER_SESSION
[.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
Joe.User@mycompantions.com" APP="Ignio Uat OAG"
APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com"
RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
10.44.65.38#0122021-03-23T16:47:20.708-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
UserName=Joe.User@mycompantions.com
SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompan'
*escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
[.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
Joe.User@mycompantions.com" APP="Ignio Uat OAG"
APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com"
RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
10.44.65.38#0122021-03-23T16:47:20.708-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
UserName=Joe.User@mycompantions.com
SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompan'
*rawmsg*: '03-23T16:47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
[.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
Joe.User@mycompantions.com" APP="Ignio Uat OAG"
APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com"
RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38"
USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"]
SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
10.44.65.38#0122021-03-23T16:47:20.708-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO
USER_AUTHZ [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompantions.com" RESOURCE="/_dash-dependencies"
METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio
Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
Joe.User@mycompantions.com RelayDomain=apigniodashboard-uat.mycompany.com
UserName=Joe.User@mycompantions.com
SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO
USER_SESSION [.SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompan'
By inspecting the rawmsg, I can see that field four (space-delimited)
indicates this is the ACCESS log. So if I were able to extract the log
identifier from the msg, I could then write all access logs to the same
daily file. There are other formats as well from the same device but the
idea is the same.
*Scott Slattery*
*Sr. Systems & Cloud Architect*
*Cloud, Compute, Information & Architecture Team*
motorolasolutions.com
*O: 602.529.8226*
*E*: Scott.Slattery@MotorolaSolutions.com
On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm> wrote:
> the source logfile name is not included in the payload by the syslog
spec.
> It
> may be in the case of your appliance, but we would need to see a sample
> log to
> understand ho to parse it.
>
> based on your template, you are using app-name, which may be listed
> separtely if
> it's a RFC5424 format log, or may be part of the syslog tag if it's a
> RFC3164
> format log over the wire (neither format has a way to specify a
source log
> file
> by default)
>
> you can look at
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e= > and see the *-cc
> options that you could apply to the app-name to eliminate control
> characters.
>
> Again, we really need to see the original log message to understand
what's
> what.
> Please log it with the templateRSYSLOG_DebugFormat so we can see exactly
> what is
> sent over the wire and how rsyslog has parsed it.
>
> David Lang
>
> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
> wrote:
>
> > Date: Tue, 23 Mar 2021 16:05:45 -0700
> > From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com>
> > To: John Chivian <jchivian@chivian.com>
> > Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
> > rsyslog-users <rsyslog@lists.adiscon.com>
> > Subject: Re: [rsyslog] Altering forwarded logfile names
> >
> > Thanks, John, let me try to clarify what I mean.
> >
> > Normally when I forward from a remote server to the central log
server, I
> > can include a tag that can then be used to determine the file name
I want
> > on the central server. Since I have no real way to include this tag
from
> > the appliance, this is not an option.
> >
> > I'm looking for a way of inspecting the incoming packets to determining
> the
> > source logfile name (which is included in the payload) and use that
> > filename on the target central server. Since there are multiple logs
> being
> > sent (access, audit, monitor, etc.), I'd like to segregate these into
> their
> > own files. I'm already using a template with the host information to
> > dynamically create the file names. I just don't know how I can go
beyond
> > this to also include the source logname.
> >
> > Here's the template I'm using. It works for all other hosts where I can
> > configure the tag but I get garbage names from the appliance. I had
hoped
> > that the appliance included some standard syslog tags but it
doesn't seem
> > so.
> >
> > template(name="DynRemoteLogFile" type="string"
> >
>
string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
> >
> > *Scott Slattery*
> >
> > *Sr. Systems & Cloud Architect*
> >
> > *Cloud, Compute, Information & Architecture Team*
> >
> > motorolasolutions.com
> >
> > *O: 602.529.8226*
> >
> > *E*: Scott.Slattery@MotorolaSolutions.com
> >
> >
> >
> >
> > On Tue, Mar 23, 2021 at 3:30 PM John Chivian <jchivian@chivian.com>
> wrote:
> >
> >> Your use of the term “file name” is confusing. When senders deliver to
> >> rsyslog over the network there is no exchange of files or filenames,
> only
> >> packets of information. Those packets are expected to be in a format
> that
> >> syslog understands such that useful information (header elements and
> >> message body) may be parsed from them. If you as the rsyslog admin
> choose
> >> to use some of that header information to compose filenames for output
> >> files, then yes you are sort of at the mercy of the senders content
> >> (especially if the sender doesn’t follow the syslog rules). However,
> there
> >> are functions in the advanced syntax that can be used to perform the
> type
> >> of character replacements you’re talking about.
> >>
> >> It is common practice to use the syslog header/rsyslog property
element
> >> called “hostname” for just such purposes. Is this what you’re talking
> >> about? You’d have to provide your configuration for real analysis, at
> >> least the part you perceive to be responsible for the problem.
> >>
> >> Regards,
> >>
> >>
> >>
> >> > On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
> >> rsyslog@lists.adiscon.com> wrote:
> >> >
> >> > I have a configured central log collector using rsyslog. A few
of the
> >> > devices forwarding their logs are appliances that have no
> configuration
> >> > options other than the IP forwarding address and protocol. I cannot
> >> control
> >> > what file names are being sent.
> >> >
> >> > Unfortunately, they are sending unintelligible file names with
> characters
> >> > that normally would be escaped. Is there any way I can control or
> alter
> >> the
> >> > incoming file name to normalize it to avoid these odd characters?
> >> >
> >> > For example, could I establish a character map that maps the
unallowed
> >> > character to something acceptable?
> >> >
> >> > thanks,
> >> >
> >> > *Scott Slattery*
> >> >
> >> > *Sr. Systems & Cloud Architect*
> >> >
> >> > *Cloud, Compute, Information & Architecture Team*
> >> >
> >> > motorolasolutions.com
> >> >
> >> > *O: 602.529.822*
> >> >
> >> > *E*: Scott.Slattery@MotorolaSolutions.com
> >> >
> >> > --
> >> >
> >> >
> >> > *For more information on how and why we collect your personal
> >> > information, please visit our Privacy Policy
> >> > <
> >>
>
https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement > >> >.*
> >> > _______________________________________________
> >> > rsyslog mailing list
> >> >
> >>
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e= > >> >
> >>
>
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e= > >> > What's up with rsyslog? Follow
> >>
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e= > >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> DON'T LIKE THAT.
> >>
> >>
> >
> > --
> >
> >
> > *For more information on how and why we collect your personal
> > information, please visit our Privacy Policy
> > <
>
https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement > >.*
> > _______________________________________________
> > rsyslog mailing list
> >
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e= > >
>
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e= > > What's up with rsyslog? Follow
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e= > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
--
*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<
https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow
https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow
https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.