Mailing List Archive: Altering forwarded logfile names

Re: Altering forwarded logfile names [ In reply to ]

Mar 25, 2021, 4:19 PM

Post #26 of 30 (545 views)

Thanks David - I'll try to upgrade my rsyslog version and see if that gives
any relief.

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: Scott.Slattery@MotorolaSolutions.com

On Thu, Mar 25, 2021 at 4:17 PM David Lang <david@lang.hm> wrote:

> I don't think this is the problem, but 8.24 is 4+ years old now (releaseed
> Jan
> 2017, although the version maintained by RedHat and Amazon has some
> bugfixes
> backported), can you try with a current version (8.2102 or 8.2012) and see
> if
> you still have the problem. I don't remember of hearing about a similar
> problem
> several years ago, but since we don't know what patches that version has
> compared to the version the commmunity released, it's hard to say. There
> are a
> LOT of bugfixes and new features introduced over that time.
>
> thanks for the config, I don't see anything odd in it.
>
> David Lang
>
> On Thu, 25 Mar 2021, Scott Slattery
> wrote:
>
> > Date: Thu, 25 Mar 2021 15:24:23 -0700
> > From: Scott Slattery <scott.slattery@motorolasolutions.com>
> > To: David Lang <david@lang.hm>
> > Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
> > rsyslog-users <rsyslog@lists.adiscon.com>, mariusz.kruk@safecomp.com
> > Subject: Re: [rsyslog] Altering forwarded logfile names
> >
> > *Rsyslogd version:*
> > rsyslogd 8.24.0-52.amzn2, compiled with:
> > PLATFORM: x86_64-koji-linux-gnu
> > PLATFORM (lsb_release -d):
> > FEATURE_REGEXP: Yes
> > GSSAPI Kerberos 5 support: Yes
> > FEATURE_DEBUG (debug build, slow code): No
> > 32bit Atomic operations supported: Yes
> > 64bit Atomic operations supported: Yes
> > memory allocator: system default
> > Runtime Instrumentation (slow code): No
> > uuid support: Yes
> > Number of Bits in RainerScript integers: 64
> >
> > Here is my config:
> > $MaxMessageSize 64k
> > $EscapeControlCharactersOnReceive on
> >
> > module(load="imuxsock")
> > module(load="imjournal" StateFile="imjournal.state")
> > module(load="imklog")
> > module(load="immark")
> > module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat"
> > DirCreateMode="0705" FileCreateMode="0704")
> > module(load="impstats" interval="600" log.syslog="off" severity="7"
> > log.file="/var/log/impstats.log")
> >
> > $DynaFileCacheSize 100
> > global(workDirectory="/var/lib/rsyslog" localHostname="cloudlogmaster01")
> >
> > $MainMsgQueueSize 200000
> > $MainMsgQueueHighWaterMark 120000
> > $MainMsgQueueDiscardMark 160000
> > $MainMsgQueueWorkerThreads 100
> > $RulesetCreateMainQueue on
> >
> > $imjournalRatelimitInterval 0
> > $imjournalRatelimitBurst 0
> > #### RULES ####
> >
> > template(name="debug" type="string" string="Debug line with all
> > properties:\nFROMHOST: '%FROMHOST%'\n, FROMHOST-IP: '%fromhost-ip%'\n,
> > HOSTNAME: '%HOSTNAME%'\n, PRI: %PRI%,\nSYSLOGTAG '%syslogtag%'\n,
> > PROGRAMNAME: '%programname%'\n, APP-NAME: '%app-name%'\n, PROCID:
> > '%PROCID%'\n, MSGID: '%MSGID%'\n, TIMESTAMP: '%TIMESTAMP%'\n,
> > STRUCTURED-DATA: '%STRUCTURED-DATA%'\n, msg: '%msg%'\nRAWMSG:
> > '%rawmsg%',\n\nRAWMSG-AFTER-PRI: '%rawmsg-after-pri%'\nJSONMESG:
> > '%jsonmesg%'\n")
> >
> > template(name="DynRemoteLogFile" type="string"
> >
> string="/splunklog/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
> >
> > template(name="MalformedMsgFormatter" type="string"
> string="%timegenerated%
> > %fromhost% %rawmsg:::drop-last-lf%\n")
> > template(name="DynRemoteMalformed" type="string"
> >
> string="/splunklog/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-oag-audit.log")
> >
> > template(name="DynRemoteLogFile" type="list") {
> > constant(value="/splunklog/remote/")
> > property(name="FROMHOST")
> > constant(value="-")
> > property(name="FROMHOST-IP")
> > constant(value="/")
> > property(name="$YEAR")
> > constant(value="-")
> > property(name="$MONTH")
> > constant(value="-")
> > property(name="$DAY")
> > constant(value="-")
> > property(name="syslogtag" position.from="1" position.to="32")
> > constant(value="-")
> > property(name="app-name" controlcharacters="drop")
> > constant(value=".log")
> > }
> >
> > ruleset(name="RemoteServer"){
> > if ($fromhost-ip startswith '10.41.102') then {
> > action(type="omfile" dynafile="DynRemoteMalformed"
> > template="MalformedMsgFormatter")
> > stop
> > }
> >
> > action(type="omfile" dynaFile="DynRemoteLogFile")
> > }
> >
> >
> > # module imtcp
> > module(load="imtcp" KeepAlive="on")
> > input(type="imtcp" port="10514" ruleset="RemoteServer")
> >
> >
> > #kern.* /dev/console
> > *.info;mail.none;authpriv.none;cron.none /var/log/messages
> > authpriv.* /var/log/secure
> > mail.* /var/log/maillog
> > cron.* /var/log/cron
> > *.emerg :omusrmsg:*
> > uucp,news.crit /var/log/spooler
> > local7.* /var/log/boot.log
> >
> > *Scott Slattery*
> >
> > *Sr. Systems & Cloud Architect*
> >
> > *Cloud, Compute, Information & Architecture Team*
> >
> > motorolasolutions.com
> >
> > *O: 602.529.8226*
> >
> > *E*: Scott.Slattery@MotorolaSolutions.com
> >
> >
> >
> >
> > On Thu, Mar 25, 2021 at 3:20 PM David Lang <david@lang.hm> wrote:
> >
> >> what version of rsyslog are you running. can you post your full config?
> >>
> >> if you are receiving via TCP and it's not splitting the logs based on
> >> newlines,
> >> something very odd is happening.
> >>
> >> David Lang
> >>
> >> On Thu, 25 Mar 2021, Scott Slattery wrote:
> >>
> >>> Date: Thu, 25 Mar 2021 15:07:22 -0700
> >>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> >>> To: David Lang <david@lang.hm>
> >>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
> >>> rsyslog-users <rsyslog@lists.adiscon.com>,
> mariusz.kruk@safecomp.com
> >>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>
> >>> So I'm looking at the pcap and on the first log record the header
> >>> information looks fine and the record terminated by 0x0A line feed.
> >>>
> >>> When I look at the record containing this record written to the rsyslog
> >>> file, the record length is 65586 (64k) long which is what I think you
> >> were
> >>> referring to earlier. Why would rsyslog write such a long record? I
> >>> initially thought my data may be getting truncated so I updated
> >>> '$MaxMessageSize 64k'. Any suggestion how I fix this behavior.
> >>>
> >>> I can confirm that each log record in the pcap is terminated by 0x0A
> and
> >>> looks in tact so must be something I need to do on rsyslog to fix this.
> >>>
> >>> *Scott Slattery*
> >>>
> >>> *Sr. Systems & Cloud Architect*
> >>>
> >>> *Cloud, Compute, Information & Architecture Team*
> >>>
> >>> motorolasolutions.com
> >>>
> >>> *O: 602.529.8226*
> >>>
> >>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Mar 25, 2021 at 1:15 PM David Lang <david@lang.hm> wrote:
> >>>
> >>>> you may want to capture with -X so that it decodes it into hex and you
> >> can
> >>>> see
> >>>> newlines vs linefeeds
> >>>>
> >>>> David Lang
> >>>>
> >>>> On Thu, 25 Mar 2021, Scott Slattery wrote:
> >>>>
> >>>>> Date: Thu, 25 Mar 2021 12:47:43 -0700
> >>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> >>>>> To: David Lang <david@lang.hm>
> >>>>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
> >>>>> rsyslog-users <rsyslog@lists.adiscon.com>,
> >> mariusz.kruk@safecomp.com
> >>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>>>
> >>>>> Thanks David, I just grabbed my first capture and am looking at it.
> I'm
> >>>>> using TCP vs UDP but your comment makes sense. I'll certainly share
> >> with
> >>>>> you my observations once I have some data to refer to. This has been
> a
> >>>>> learning experience. It's also the first time I've ever seen such an
> >>>> issue
> >>>>> with rsyslog so it's a bit of a puzzle.
> >>>>>
> >>>>> *Scott Slattery*
> >>>>>
> >>>>> *Sr. Systems & Cloud Architect*
> >>>>>
> >>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>
> >>>>> motorolasolutions.com
> >>>>>
> >>>>> *O: 602.529.8226*
> >>>>>
> >>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Mar 25, 2021 at 12:32 PM David Lang <david@lang.hm> wrote:
> >>>>>
> >>>>>> the rawmsg field in the debugformat output shows exactly what
> rsyslog
> >> is
> >>>>>> seeing.
> >>>>>>
> >>>>>> the reason I asked you to check multiple entries is that if rsyslog
> >> does
> >>>>>> not see
> >>>>>> the separator (due to either multiple messages in one UDP packet, or
> >>>>>> missing
> >>>>>> newlines in a TCP stream) it will combine what are intended as
> >> multiple
> >>>>>> messages
> >>>>>> together into one message as far as rsyslog is concerned, and only
> >> break
> >>>>>> it into
> >>>>>> multiple messages when it hits the configured maxmessagesize size.
> >> That
> >>>>>> breaks
> >>>>>> things at arbitrary points in the message, so as you look at
> multiple
> >>>>>> messages,
> >>>>>> they will not all start the same way.
> >>>>>>
> >>>>>> a tcpdump -s0 -A -n (with the appropriate port/ip filter) will also
> >> show
> >>>>>> exactly
> >>>>>> what's happening over the wire.
> >>>>>>
> >>>>>> David Lang
> >>>>>>
> >>>>>> On Thu, 25 Mar 2021, Scott Slattery wrote:
> >>>>>>
> >>>>>>> Date: Thu, 25 Mar 2021 12:23:03 -0700
> >>>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> >>>>>>> To: Rainer Gerhards <rgerhards@hq.adiscon.com>
> >>>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
> >>>> david@lang.hm
> >>>>>>> ,
> >>>>>>> mariusz.kruk@safecomp.com
> >>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>>>>>
> >>>>>>> Hey Guys, before I consume more of your time, I'm capturing the
> >> packets
> >>>>>>> from the source device to determine if any of the payload is lost
> on
> >>>> its
> >>>>>>> way to the rsyslog collector. After I analyze further, I'll share
> my
> >>>>>>> observations. I'm hoping nothing is getting dropped over the wire.
> >>>> Thanks
> >>>>>>> for your patience and help so far. I expect to have additional
> >> details
> >>>>>>> shortly.
> >>>>>>>
> >>>>>>> *Scott Slattery*
> >>>>>>>
> >>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>
> >>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>
> >>>>>>> motorolasolutions.com
> >>>>>>>
> >>>>>>> *O: 602.529.8226*
> >>>>>>>
> >>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Mar 25, 2021 at 12:51 AM Rainer Gerhards <
> >>>>>> rgerhards@hq.adiscon.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Sorry if I didn't catch it, but did you post the configuration?
> If
> >>>>>>>> not, it would probably be useful.
> >>>>>>>>
> >>>>>>>> Looking at the message properties, this looks like the message was
> >>>>>>>> received via UDP but had a LF inside it which was intended as a
> >> frame
> >>>>>>>> separator. If so, that is another bug in the vendor's
> >> implementation.
> >>>>>>>> Framing is used for TCP, but not UDP. For UDP it is strictly one
> >>>>>>>> message per package.
> >>>>>>>>
> >>>>>>>> HTH,
> >>>>>>>> Rainer
> >>>>>>>>
> >>>>>>>> El mié, 24 mar 2021 a las 22:23, Scott Slattery via rsyslog
> >>>>>>>> (<rsyslog@lists.adiscon.com>) escribió:
> >>>>>>>>>
> >>>>>>>>> From what I can discern since there are lots of transactions,
> there
> >>>>>> seem
> >>>>>>>> to
> >>>>>>>>> be multiple payload formats. For all that I have so far analyzed,
> >> the
> >>>>>>>> date
> >>>>>>>>> truncation issue is the same regardless of the payload received.
> >> Like
> >>>>>>>> you,
> >>>>>>>>> I've never used mmnormalize but I will look at it as an option.
> >> I've
> >>>>>>>>> reached out to the product support team to better understand if
> >> there
> >>>>>> is
> >>>>>>>> a
> >>>>>>>>> appliance-side misconfiguration and something that can be
> corrected
> >>>>>> there
> >>>>>>>>> or if I need to make adjustments on the rsyslog side to clean up
> >> the
> >>>>>>>>> records.
> >>>>>>>>>
> >>>>>>>>> What I meant is that there are multiple message formats (payload
> >>>> data)
> >>>>>>>> but
> >>>>>>>>> there seems to be a consistent truncation of the date field just
> as
> >>>> you
> >>>>>>>> saw
> >>>>>>>>> in the app-name tag field.
> >>>>>>>>>
> >>>>>>>>> I appreciate your ongoing feedback and help. This issue is new to
> >> me
> >>>>>> and
> >>>>>>>>> will need a creative solution if the vendor has no answers.
> >>>>>>>>>
> >>>>>>>>> Many thanks,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> *Scott Slattery*
> >>>>>>>>>
> >>>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>>
> >>>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>>
> >>>>>>>>> motorolasolutions.com
> >>>>>>>>>
> >>>>>>>>> *O: 602.529.8226*
> >>>>>>>>>
> >>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, Mar 24, 2021 at 2:14 PM David Lang <david@lang.hm>
> wrote:
> >>>>>>>>>
> >>>>>>>>>> when you say 'each is incorrect but the same format'
> >>>>>>>>>>
> >>>>>>>>>> does that mean that every log has the year missing? or that
> every
> >>>> log
> >>>>>>>> is
> >>>>>>>>>> combining the logs together?
> >>>>>>>>>>
> >>>>>>>>>> I'll note that it's possible to define a custom parser using
> the
> >>>>>>>>>> mmnormalize
> >>>>>>>>>> library and add it to the parsing stack. I helped define the
> need
> >>>> for
> >>>>>>>> such
> >>>>>>>>>> a
> >>>>>>>>>> capability, but haven't used it yet.
> >>>>>>>>>>
> >>>>>>>>>> David Lang
> >>>>>>>>>>
> >>>>>>>>>> On Wed, 24 Mar 2021, Scott Slattery wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Date: Wed, 24 Mar 2021 08:58:07 -0700
> >>>>>>>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> >>>>>>>>>>> To: mariusz.kruk@safecomp.com
> >>>>>>>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
> >>>>>>>> david@lang.hm
> >>>>>>>>>>>
> >>>>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>>>>>>>>>
> >>>>>>>>>>> Yes, all the logs are coming in this way. There are three
> >> separate
> >>>>>>>> logs
> >>>>>>>>>>> being sent and each is incorrect in its own right but the same
> >>>> format
> >>>>>>>>>>> behavior is consistent across each of the three logs. I will
> look
> >>>>>>>> further
> >>>>>>>>>>> into the suggestions you have provided and have already reached
> >> out
> >>>>>>>> to
> >>>>>>>>>> the
> >>>>>>>>>>> appliance support team and will raise the nonconformance with
> the
> >>>>>>>> RFC.
> >>>>>>>>>>>
> >>>>>>>>>>> This does provide me some valuable guidance and I thank each of
> >> you
> >>>>>>>> for
> >>>>>>>>>>> your feedback.
> >>>>>>>>>>>
> >>>>>>>>>>> *Scott Slattery*
> >>>>>>>>>>>
> >>>>>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>>>>
> >>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>>>>
> >>>>>>>>>>> motorolasolutions.com
> >>>>>>>>>>>
> >>>>>>>>>>> *O: 602.529.8226*
> >>>>>>>>>>>
> >>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com>
> >>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> There's something strange with this log entry. Notice that it
> >>>> looks
> >>>>>>>> as
> >>>>>>>>>> if
> >>>>>>>>>>>> it were more than one log entries joined with LF character
> >>>> (escaped
> >>>>>>>> on
> >>>>>>>>>>>> input as #012). And all "further" entries start with a - still
> >>>>>>>>>>>> nonconformant to the RFC but "fuller" - timestamp containing
> >>>>>>>> year.It's
> >>>>>>>>>> the
> >>>>>>>>>>>> leading event that seems to be trimmed in the front somehow.
> Do
> >>>> all
> >>>>>>>>>> events
> >>>>>>>>>>>> arrive like this?Wys?ano z telefonu Samsung<div>
> >>>>>>>>>>>> </div><div>
> >>>>>>>>>>>> </div><div>-------- Original message
> >>>>>>>>>>>> --------</div><div>From: Scott Slattery via rsyslog <
> >>>>>>>>>>>> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021 01:52
> >>>>>>>>>>>> (GMT+01:00) </div><div>To: David Lang <david@lang.hm>
> >>>>>>>> </div><div>Cc:
> >>>>>>>>>>>> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott
> >>>>>>>> Slattery
> >>>>>>>>>> via
> >>>>>>>>>>>> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re:
> >>>>>>>> [rsyslog]
> >>>>>>>>>>>> Altering forwarded logfile names </div><div>
> >>>>>>>>>>>> </div>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi David, fortunately I had already done this. I'm including
> an
> >>>>>>>> actual
> >>>>>>>>>> log
> >>>>>>>>>>>> entry but have anonymized the data to keep the actual user and
> >>>> email
> >>>>>>>>>>>> address confidential:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Debug line with all properties:
> >>>>>>>>>>>> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
> >>>>>>>>>>>> , fromhost-ip: '10.41.102.143'
> >>>>>>>>>>>> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
> >>>>>>>>>>>> , PRI: 13,
> >>>>>>>>>>>> syslogtag '03-23T16:'
> >>>>>>>>>>>> , programname: '03-23T16'
> >>>>>>>>>>>> , APP-NAME: '03-23T16'
> >>>>>>>>>>>> , PROCID: '-'
> >>>>>>>>>>>> , MSGID: '-'
> >>>>>>>>>>>> , TIMNESTAMP: 'Mar 23 21:47:13'
> >>>>>>>>>>>> , STRUCTURED-DATA: '-'
> >>>>>>>>>>>> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> >>>>>>>>>> ACCESS_GATEWAY
> >>>>>>>>>>>> ACCESS AUTHZ SESSION INFO USER_SESSION
> >>>>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>> SUBJECT="
> >>>>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >>>>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>>>>>> apigniodashboard-uat.mycompany.com"
> >>>>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> >>>>>>>>>> REMOTE_IP="10.44.65.38"
> >>>>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >>>>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>>>>>> Safari/537.36"]
> >>>>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >>>>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>> POLICY
> >>>>>>>>>> INFO
> >>>>>>>>>>>> USER_AUTHZ
> >>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
> >>>> RESOURCE="/_dash-dependencies"
> >>>>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED"
> DURATION="0"
> >>>>>>>>>> APP="Ignio
> >>>>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW"
> REASON="N/A
> >> -
> >>>>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
> >>>> oag_username=
> >>>>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
> >>>>>>>>>> apigniodashboard-uat.mycompany.com
> >>>>>>>>>>>> UserName=Joe.User@mycompantions.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >>>>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
> >> Mac
> >>>>>>>> OS X
> >>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> >>>> Chrome/88.0.4324.192
> >>>>>>>>>>>> Safari/537.36 creationTime=1616536016811
> >>>> maxInactiveInterval=3600000
> >>>>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >>>>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
> >> Intel
> >>>>>>>> Mac
> >>>>>>>>>> OS X
> >>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> >>>> Chrome/88.0.4324.192
> >>>>>>>>>>>> Safari/537.36"] allow access to
> >>>>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
> >>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>>>>>> SESSION
> >>>>>>>>>> INFO
> >>>>>>>>>>>> USER_SESSION
> >>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>>>>>> SUBJECT="Joe.User@mycompan'
> >>>>>>>>>>>> *escaped msg*: '47:20.708-05:00
> >> AUSE1OAGATST02.aws.mycompany.com
> >>>>>>>>>>>> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
> >>>>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>> SUBJECT="
> >>>>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >>>>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>>>>>> apigniodashboard-uat.mycompany.com"
> >>>>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> >>>>>>>>>> REMOTE_IP="10.44.65.38"
> >>>>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >>>>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>>>>>> Safari/537.36"]
> >>>>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >>>>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>> POLICY
> >>>>>>>>>> INFO
> >>>>>>>>>>>> USER_AUTHZ
> >>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
> >>>> RESOURCE="/_dash-dependencies"
> >>>>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED"
> DURATION="0"
> >>>>>>>>>> APP="Ignio
> >>>>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW"
> REASON="N/A
> >> -
> >>>>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
> >>>> oag_username=
> >>>>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
> >>>>>>>>>> apigniodashboard-uat.mycompany.com
> >>>>>>>>>>>> UserName=Joe.User@mycompantions.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >>>>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
> >> Mac
> >>>>>>>> OS X
> >>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> >>>> Chrome/88.0.4324.192
> >>>>>>>>>>>> Safari/537.36 creationTime=1616536016811
> >>>> maxInactiveInterval=3600000
> >>>>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >>>>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
> >> Intel
> >>>>>>>> Mac
> >>>>>>>>>> OS X
> >>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> >>>> Chrome/88.0.4324.192
> >>>>>>>>>>>> Safari/537.36"] allow access to
> >>>>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
> >>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>>>>>> SESSION
> >>>>>>>>>> INFO
> >>>>>>>>>>>> USER_SESSION
> >>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>>>>>> SUBJECT="Joe.User@mycompan'
> >>>>>>>>>>>> *rawmsg*: '03-23T16:47:20.708-05:00
> >>>>>>>> AUSE1OAGATST02.aws.mycompany.com
> >>>>>>>>>>>> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
> >>>>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>> SUBJECT="
> >>>>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >>>>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>>>>>> apigniodashboard-uat.mycompany.com"
> >>>>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> >>>>>>>>>> REMOTE_IP="10.44.65.38"
> >>>>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >>>>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>>>>>> Safari/537.36"]
> >>>>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >>>>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>> POLICY
> >>>>>>>>>> INFO
> >>>>>>>>>>>> USER_AUTHZ
> >>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
> >>>> RESOURCE="/_dash-dependencies"
> >>>>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED"
> DURATION="0"
> >>>>>>>>>> APP="Ignio
> >>>>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW"
> REASON="N/A
> >> -
> >>>>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
> >>>> oag_username=
> >>>>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
> >>>>>>>>>> apigniodashboard-uat.mycompany.com
> >>>>>>>>>>>> UserName=Joe.User@mycompantions.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >>>>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
> >> Mac
> >>>>>>>> OS X
> >>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> >>>> Chrome/88.0.4324.192
> >>>>>>>>>>>> Safari/537.36 creationTime=1616536016811
> >>>> maxInactiveInterval=3600000
> >>>>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >>>>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
> >> Intel
> >>>>>>>> Mac
> >>>>>>>>>> OS X
> >>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> >>>> Chrome/88.0.4324.192
> >>>>>>>>>>>> Safari/537.36"] allow access to
> >>>>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
> >>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>>>>>> SESSION
> >>>>>>>>>> INFO
> >>>>>>>>>>>> USER_SESSION
> >>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>>>>>> SUBJECT="Joe.User@mycompan'
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> By inspecting the rawmsg, I can see that field four
> >>>>>>>> (space-delimited)
> >>>>>>>>>>>> indicates this is the ACCESS log. So if I were able to extract
> >> the
> >>>>>>>> log
> >>>>>>>>>>>> identifier from the msg, I could then write all access logs to
> >> the
> >>>>>>>> same
> >>>>>>>>>>>> daily file. There are other formats as well from the same
> device
> >>>>>>>> but the
> >>>>>>>>>>>> idea is the same.
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Scott Slattery*
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>>>>>
> >>>>>>>>>>>> motorolasolutions.com
> >>>>>>>>>>>>
> >>>>>>>>>>>> *O: 602.529.8226*
> >>>>>>>>>>>>
> >>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm>
> >> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> the source logfile name is not included in the payload by the
> >>>>>>>> syslog
> >>>>>>>>>>>> spec.
> >>>>>>>>>>>>> It
> >>>>>>>>>>>>> may be in the case of your appliance, but we would need to
> see
> >> a
> >>>>>>>> sample
> >>>>>>>>>>>>> log to
> >>>>>>>>>>>>> understand ho to parse it.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> based on your template, you are using app-name, which may be
> >>>> listed
> >>>>>>>>>>>>> separtely if
> >>>>>>>>>>>>> it's a RFC5424 format log, or may be part of the syslog tag
> if
> >>>>>>>> it's a
> >>>>>>>>>>>>> RFC3164
> >>>>>>>>>>>>> format log over the wire (neither format has a way to
> specify a
> >>>>>>>> source
> >>>>>>>>>>>> log
> >>>>>>>>>>>>> file
> >>>>>>>>>>>>> by default)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> you can look at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
> >>>>>>>>>>>>> and see the *-cc
> >>>>>>>>>>>>> options that you could apply to the app-name to eliminate
> >> control
> >>>>>>>>>>>>> characters.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Again, we really need to see the original log message to
> >>>> understand
> >>>>>>>>>>>> what's
> >>>>>>>>>>>>> what.
> >>>>>>>>>>>>> Please log it with the templateRSYSLOG_DebugFormat so we can
> >> see
> >>>>>>>>>> exactly
> >>>>>>>>>>>>> what is
> >>>>>>>>>>>>> sent over the wire and how rsyslog has parsed it.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> David Lang
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
> >>>>>>>>>>>>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com
> >
> >>>>>>>>>>>>>> To: John Chivian <jchivian@chivian.com>
> >>>>>>>>>>>>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
> >>>>>>>>>>>>>> rsyslog-users <rsyslog@lists.adiscon.com>
> >>>>>>>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks, John, let me try to clarify what I mean.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Normally when I forward from a remote server to the central
> >> log
> >>>>>>>>>>>> server, I
> >>>>>>>>>>>>>> can include a tag that can then be used to determine the
> file
> >>>>>>>> name I
> >>>>>>>>>>>> want
> >>>>>>>>>>>>>> on the central server. Since I have no real way to include
> >> this
> >>>>>>>> tag
> >>>>>>>>>>>> from
> >>>>>>>>>>>>>> the appliance, this is not an option.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'm looking for a way of inspecting the incoming packets to
> >>>>>>>>>> determining
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>> source logfile name (which is included in the payload) and
> use
> >>>>>>>> that
> >>>>>>>>>>>>>> filename on the target central server. Since there are
> >> multiple
> >>>>>>>> logs
> >>>>>>>>>>>>> being
> >>>>>>>>>>>>>> sent (access, audit, monitor, etc.), I'd like to segregate
> >> these
> >>>>>>>> into
> >>>>>>>>>>>>> their
> >>>>>>>>>>>>>> own files. I'm already using a template with the host
> >>>> information
> >>>>>>>> to
> >>>>>>>>>>>>>> dynamically create the file names. I just don't know how I
> can
> >>>> go
> >>>>>>>>>>>> beyond
> >>>>>>>>>>>>>> this to also include the source logname.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Here's the template I'm using. It works for all other hosts
> >>>> where
> >>>>>>>> I
> >>>>>>>>>> can
> >>>>>>>>>>>>>> configure the tag but I get garbage names from the
> appliance.
> >> I
> >>>>>>>> had
> >>>>>>>>>>>> hoped
> >>>>>>>>>>>>>> that the appliance included some standard syslog tags but it
> >>>>>>>> doesn't
> >>>>>>>>>>>> seem
> >>>>>>>>>>>>>> so.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> template(name="DynRemoteLogFile" type="string"
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *Scott Slattery*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> motorolasolutions.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *O: 602.529.8226*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <
> >>>>>>>> jchivian@chivian.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Your use of the term “file name” is confusing. When
> senders
> >>>>>>>> deliver
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>> rsyslog over the network there is no exchange of files or
> >>>>>>>> filenames,
> >>>>>>>>>>>>> only
> >>>>>>>>>>>>>>> packets of information. Those packets are expected to be
> in
> >> a
> >>>>>>>> format
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>>> syslog understands such that useful information (header
> >>>> elements
> >>>>>>>> and
> >>>>>>>>>>>>>>> message body) may be parsed from them. If you as the
> rsyslog
> >>>>>>>> admin
> >>>>>>>>>>>>> choose
> >>>>>>>>>>>>>>> to use some of that header information to compose filenames
> >> for
> >>>>>>>>>> output
> >>>>>>>>>>>>>>> files, then yes you are sort of at the mercy of the senders
> >>>>>>>> content
> >>>>>>>>>>>>>>> (especially if the sender doesn’t follow the syslog rules).
> >>>>>>>> However,
> >>>>>>>>>>>>> there
> >>>>>>>>>>>>>>> are functions in the advanced syntax that can be used to
> >>>> perform
> >>>>>>>> the
> >>>>>>>>>>>>> type
> >>>>>>>>>>>>>>> of character replacements you’re talking about.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> It is common practice to use the syslog header/rsyslog
> >> property
> >>>>>>>>>>>> element
> >>>>>>>>>>>>>>> called “hostname” for just such purposes. Is this what
> >> you’re
> >>>>>>>>>> talking
> >>>>>>>>>>>>>>> about? You’d have to provide your configuration for real
> >>>>>>>> analysis,
> >>>>>>>>>> at
> >>>>>>>>>>>>>>> least the part you perceive to be responsible for the
> >> problem.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
> >>>>>>>>>>>>>>> rsyslog@lists.adiscon.com> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I have a configured central log collector using rsyslog. A
> >> few
> >>>>>>>> of
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> devices forwarding their logs are appliances that have no
> >>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>>> options other than the IP forwarding address and
> protocol. I
> >>>>>>>> cannot
> >>>>>>>>>>>>>>> control
> >>>>>>>>>>>>>>>> what file names are being sent.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Unfortunately, they are sending unintelligible file names
> >> with
> >>>>>>>>>>>>> characters
> >>>>>>>>>>>>>>>> that normally would be escaped. Is there any way I can
> >> control
> >>>>>>>> or
> >>>>>>>>>>>>> alter
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> incoming file name to normalize it to avoid these odd
> >>>>>>>> characters?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> For example, could I establish a character map that maps
> the
> >>>>>>>>>>>> unallowed
> >>>>>>>>>>>>>>>> character to something acceptable?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> thanks,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> *Scott Slattery*
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> motorolasolutions.com
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> *O: 602.529.822*
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> *For more information on how and why we collect your
> >> personal
> >>>>>>>>>>>>>>>> information, please visit our Privacy Policy
> >>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>>>>>>>>>>> .*
> >>>>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>>>> rsyslog mailing list
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
> >>>>>>>>>>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
> >>>>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
> ARCHIVED
> >>>> by
> >>>>>>>> a
> >>>>>>>>>>>>> myriad
> >>>>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> >> POST
> >>>>>>>> if
> >>>>>>>>>> you
> >>>>>>>>>>>>>>> DON'T LIKE THAT.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *For more information on how and why we collect your
> personal
> >>>>>>>>>>>>>> information, please visit our Privacy Policy
> >>>>>>>>>>>>>> <
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>>>>>>>>> .*
> >>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>> rsyslog mailing list
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
> >>>>>>>>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
> >>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> >> by
> >>>> a
> >>>>>>>>>>>> myriad
> >>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> POST
> >>>> if
> >>>>>>>> you
> >>>>>>>>>>>>> DON'T LIKE THAT.
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> *For more information on how and why we collect your personal
> >>>>>>>>>>>> information, please visit our Privacy Policy
> >>>>>>>>>>>> <
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>>>>>>>> .*
> >>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>> rsyslog mailing list
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
> >>>>>>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
> >>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> by
> >> a
> >>>>>>>> myriad
> >>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> POST
> >> if
> >>>>>>>> you
> >>>>>>>>>>>> DON'T LIKE THAT.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> *For more information on how and why we collect your personal
> >>>>>>>>> information, please visit our Privacy Policy
> >>>>>>>>> <
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>>>> .*
> >>>>>>>>> _______________________________________________
> >>>>>>>>> rsyslog mailing list
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=7hc9b2gSfvsfQzXvsRVXqGeP4Rz_tJtrykpjqkGBIdk&e=
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=bDybhvts9KqI-WEfMHo7O40ulQK5M6Tg4tnKNRRIhE8&e=
> >>>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=zPhpcUPlK_CWLiU75_WwjyQGz5mFTiXc1yyBIfppkrI&e=
> >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>>>>> myriad
> >>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> >> you
> >>>>>>>> DON'T LIKE THAT.
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >
> >

--

*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Altering forwarded logfile names [ In reply to ]

rsyslog at lists

Mar 25, 2021, 5:03 PM

Post #27 of 30 (545 views)

Permalink

So I upgraded to the following current version and I don't see any
behavioral differences. I'll continue investigating tomorrow but again
thank you for sticking with me on this adventure.

rsyslogd 8.2102.0.master (aka 2021.02) compiled with:
PLATFORM: x86_64-redhat-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
systemd support: Yes
Config file: /etc/rsyslog.conf
PID file: /var/run/syslogd.pid
Number of Bits in RainerScript integers: 64

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: Scott.Slattery@MotorolaSolutions.com

On Thu, Mar 25, 2021 at 4:17 PM David Lang <david@lang.hm> wrote:

> I don't think this is the problem, but 8.24 is 4+ years old now (releaseed
> Jan
> 2017, although the version maintained by RedHat and Amazon has some
> bugfixes
> backported), can you try with a current version (8.2102 or 8.2012) and see
> if
> you still have the problem. I don't remember of hearing about a similar
> problem
> several years ago, but since we don't know what patches that version has
> compared to the version the commmunity released, it's hard to say. There
> are a
> LOT of bugfixes and new features introduced over that time.
>
> thanks for the config, I don't see anything odd in it.
>
> David Lang
>
> On Thu, 25 Mar 2021, Scott Slattery
> wrote:
>
> > Date: Thu, 25 Mar 2021 15:24:23 -0700
> > From: Scott Slattery <scott.slattery@motorolasolutions.com>
> > To: David Lang <david@lang.hm>
> > Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
> > rsyslog-users <rsyslog@lists.adiscon.com>, mariusz.kruk@safecomp.com
> > Subject: Re: [rsyslog] Altering forwarded logfile names
> >
> > *Rsyslogd version:*
> > rsyslogd 8.24.0-52.amzn2, compiled with:
> > PLATFORM: x86_64-koji-linux-gnu
> > PLATFORM (lsb_release -d):
> > FEATURE_REGEXP: Yes
> > GSSAPI Kerberos 5 support: Yes
> > FEATURE_DEBUG (debug build, slow code): No
> > 32bit Atomic operations supported: Yes
> > 64bit Atomic operations supported: Yes
> > memory allocator: system default
> > Runtime Instrumentation (slow code): No
> > uuid support: Yes
> > Number of Bits in RainerScript integers: 64
> >
> > Here is my config:
> > $MaxMessageSize 64k
> > $EscapeControlCharactersOnReceive on
> >
> > module(load="imuxsock")
> > module(load="imjournal" StateFile="imjournal.state")
> > module(load="imklog")
> > module(load="immark")
> > module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat"
> > DirCreateMode="0705" FileCreateMode="0704")
> > module(load="impstats" interval="600" log.syslog="off" severity="7"
> > log.file="/var/log/impstats.log")
> >
> > $DynaFileCacheSize 100
> > global(workDirectory="/var/lib/rsyslog" localHostname="cloudlogmaster01")
> >
> > $MainMsgQueueSize 200000
> > $MainMsgQueueHighWaterMark 120000
> > $MainMsgQueueDiscardMark 160000
> > $MainMsgQueueWorkerThreads 100
> > $RulesetCreateMainQueue on
> >
> > $imjournalRatelimitInterval 0
> > $imjournalRatelimitBurst 0
> > #### RULES ####
> >
> > template(name="debug" type="string" string="Debug line with all
> > properties:\nFROMHOST: '%FROMHOST%'\n, FROMHOST-IP: '%fromhost-ip%'\n,
> > HOSTNAME: '%HOSTNAME%'\n, PRI: %PRI%,\nSYSLOGTAG '%syslogtag%'\n,
> > PROGRAMNAME: '%programname%'\n, APP-NAME: '%app-name%'\n, PROCID:
> > '%PROCID%'\n, MSGID: '%MSGID%'\n, TIMESTAMP: '%TIMESTAMP%'\n,
> > STRUCTURED-DATA: '%STRUCTURED-DATA%'\n, msg: '%msg%'\nRAWMSG:
> > '%rawmsg%',\n\nRAWMSG-AFTER-PRI: '%rawmsg-after-pri%'\nJSONMESG:
> > '%jsonmesg%'\n")
> >
> > template(name="DynRemoteLogFile" type="string"
> >
> string="/splunklog/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
> >
> > template(name="MalformedMsgFormatter" type="string"
> string="%timegenerated%
> > %fromhost% %rawmsg:::drop-last-lf%\n")
> > template(name="DynRemoteMalformed" type="string"
> >
> string="/splunklog/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-oag-audit.log")
> >
> > template(name="DynRemoteLogFile" type="list") {
> > constant(value="/splunklog/remote/")
> > property(name="FROMHOST")
> > constant(value="-")
> > property(name="FROMHOST-IP")
> > constant(value="/")
> > property(name="$YEAR")
> > constant(value="-")
> > property(name="$MONTH")
> > constant(value="-")
> > property(name="$DAY")
> > constant(value="-")
> > property(name="syslogtag" position.from="1" position.to="32")
> > constant(value="-")
> > property(name="app-name" controlcharacters="drop")
> > constant(value=".log")
> > }
> >
> > ruleset(name="RemoteServer"){
> > if ($fromhost-ip startswith '10.41.102') then {
> > action(type="omfile" dynafile="DynRemoteMalformed"
> > template="MalformedMsgFormatter")
> > stop
> > }
> >
> > action(type="omfile" dynaFile="DynRemoteLogFile")
> > }
> >
> >
> > # module imtcp
> > module(load="imtcp" KeepAlive="on")
> > input(type="imtcp" port="10514" ruleset="RemoteServer")
> >
> >
> > #kern.* /dev/console
> > *.info;mail.none;authpriv.none;cron.none /var/log/messages
> > authpriv.* /var/log/secure
> > mail.* /var/log/maillog
> > cron.* /var/log/cron
> > *.emerg :omusrmsg:*
> > uucp,news.crit /var/log/spooler
> > local7.* /var/log/boot.log
> >
> > *Scott Slattery*
> >
> > *Sr. Systems & Cloud Architect*
> >
> > *Cloud, Compute, Information & Architecture Team*
> >
> > motorolasolutions.com
> >
> > *O: 602.529.8226*
> >
> > *E*: Scott.Slattery@MotorolaSolutions.com
> >
> >
> >
> >
> > On Thu, Mar 25, 2021 at 3:20 PM David Lang <david@lang.hm> wrote:
> >
> >> what version of rsyslog are you running. can you post your full config?
> >>
> >> if you are receiving via TCP and it's not splitting the logs based on
> >> newlines,
> >> something very odd is happening.
> >>
> >> David Lang
> >>
> >> On Thu, 25 Mar 2021, Scott Slattery wrote:
> >>
> >>> Date: Thu, 25 Mar 2021 15:07:22 -0700
> >>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> >>> To: David Lang <david@lang.hm>
> >>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
> >>> rsyslog-users <rsyslog@lists.adiscon.com>,
> mariusz.kruk@safecomp.com
> >>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>
> >>> So I'm looking at the pcap and on the first log record the header
> >>> information looks fine and the record terminated by 0x0A line feed.
> >>>
> >>> When I look at the record containing this record written to the rsyslog
> >>> file, the record length is 65586 (64k) long which is what I think you
> >> were
> >>> referring to earlier. Why would rsyslog write such a long record? I
> >>> initially thought my data may be getting truncated so I updated
> >>> '$MaxMessageSize 64k'. Any suggestion how I fix this behavior.
> >>>
> >>> I can confirm that each log record in the pcap is terminated by 0x0A
> and
> >>> looks in tact so must be something I need to do on rsyslog to fix this.
> >>>
> >>> *Scott Slattery*
> >>>
> >>> *Sr. Systems & Cloud Architect*
> >>>
> >>> *Cloud, Compute, Information & Architecture Team*
> >>>
> >>> motorolasolutions.com
> >>>
> >>> *O: 602.529.8226*
> >>>
> >>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Mar 25, 2021 at 1:15 PM David Lang <david@lang.hm> wrote:
> >>>
> >>>> you may want to capture with -X so that it decodes it into hex and you
> >> can
> >>>> see
> >>>> newlines vs linefeeds
> >>>>
> >>>> David Lang
> >>>>
> >>>> On Thu, 25 Mar 2021, Scott Slattery wrote:
> >>>>
> >>>>> Date: Thu, 25 Mar 2021 12:47:43 -0700
> >>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> >>>>> To: David Lang <david@lang.hm>
> >>>>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
> >>>>> rsyslog-users <rsyslog@lists.adiscon.com>,
> >> mariusz.kruk@safecomp.com
> >>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>>>
> >>>>> Thanks David, I just grabbed my first capture and am looking at it.
> I'm
> >>>>> using TCP vs UDP but your comment makes sense. I'll certainly share
> >> with
> >>>>> you my observations once I have some data to refer to. This has been
> a
> >>>>> learning experience. It's also the first time I've ever seen such an
> >>>> issue
> >>>>> with rsyslog so it's a bit of a puzzle.
> >>>>>
> >>>>> *Scott Slattery*
> >>>>>
> >>>>> *Sr. Systems & Cloud Architect*
> >>>>>
> >>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>
> >>>>> motorolasolutions.com
> >>>>>
> >>>>> *O: 602.529.8226*
> >>>>>
> >>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Mar 25, 2021 at 12:32 PM David Lang <david@lang.hm> wrote:
> >>>>>
> >>>>>> the rawmsg field in the debugformat output shows exactly what
> rsyslog
> >> is
> >>>>>> seeing.
> >>>>>>
> >>>>>> the reason I asked you to check multiple entries is that if rsyslog
> >> does
> >>>>>> not see
> >>>>>> the separator (due to either multiple messages in one UDP packet, or
> >>>>>> missing
> >>>>>> newlines in a TCP stream) it will combine what are intended as
> >> multiple
> >>>>>> messages
> >>>>>> together into one message as far as rsyslog is concerned, and only
> >> break
> >>>>>> it into
> >>>>>> multiple messages when it hits the configured maxmessagesize size.
> >> That
> >>>>>> breaks
> >>>>>> things at arbitrary points in the message, so as you look at
> multiple
> >>>>>> messages,
> >>>>>> they will not all start the same way.
> >>>>>>
> >>>>>> a tcpdump -s0 -A -n (with the appropriate port/ip filter) will also
> >> show
> >>>>>> exactly
> >>>>>> what's happening over the wire.
> >>>>>>
> >>>>>> David Lang
> >>>>>>
> >>>>>> On Thu, 25 Mar 2021, Scott Slattery wrote:
> >>>>>>
> >>>>>>> Date: Thu, 25 Mar 2021 12:23:03 -0700
> >>>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> >>>>>>> To: Rainer Gerhards <rgerhards@hq.adiscon.com>
> >>>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
> >>>> david@lang.hm
> >>>>>>> ,
> >>>>>>> mariusz.kruk@safecomp.com
> >>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>>>>>
> >>>>>>> Hey Guys, before I consume more of your time, I'm capturing the
> >> packets
> >>>>>>> from the source device to determine if any of the payload is lost
> on
> >>>> its
> >>>>>>> way to the rsyslog collector. After I analyze further, I'll share
> my
> >>>>>>> observations. I'm hoping nothing is getting dropped over the wire.
> >>>> Thanks
> >>>>>>> for your patience and help so far. I expect to have additional
> >> details
> >>>>>>> shortly.
> >>>>>>>
> >>>>>>> *Scott Slattery*
> >>>>>>>
> >>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>
> >>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>
> >>>>>>> motorolasolutions.com
> >>>>>>>
> >>>>>>> *O: 602.529.8226*
> >>>>>>>
> >>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Mar 25, 2021 at 12:51 AM Rainer Gerhards <
> >>>>>> rgerhards@hq.adiscon.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Sorry if I didn't catch it, but did you post the configuration?
> If
> >>>>>>>> not, it would probably be useful.
> >>>>>>>>
> >>>>>>>> Looking at the message properties, this looks like the message was
> >>>>>>>> received via UDP but had a LF inside it which was intended as a
> >> frame
> >>>>>>>> separator. If so, that is another bug in the vendor's
> >> implementation.
> >>>>>>>> Framing is used for TCP, but not UDP. For UDP it is strictly one
> >>>>>>>> message per package.
> >>>>>>>>
> >>>>>>>> HTH,
> >>>>>>>> Rainer
> >>>>>>>>
> >>>>>>>> El mié, 24 mar 2021 a las 22:23, Scott Slattery via rsyslog
> >>>>>>>> (<rsyslog@lists.adiscon.com>) escribió:
> >>>>>>>>>
> >>>>>>>>> From what I can discern since there are lots of transactions,
> there
> >>>>>> seem
> >>>>>>>> to
> >>>>>>>>> be multiple payload formats. For all that I have so far analyzed,
> >> the
> >>>>>>>> date
> >>>>>>>>> truncation issue is the same regardless of the payload received.
> >> Like
> >>>>>>>> you,
> >>>>>>>>> I've never used mmnormalize but I will look at it as an option.
> >> I've
> >>>>>>>>> reached out to the product support team to better understand if
> >> there
> >>>>>> is
> >>>>>>>> a
> >>>>>>>>> appliance-side misconfiguration and something that can be
> corrected
> >>>>>> there
> >>>>>>>>> or if I need to make adjustments on the rsyslog side to clean up
> >> the
> >>>>>>>>> records.
> >>>>>>>>>
> >>>>>>>>> What I meant is that there are multiple message formats (payload
> >>>> data)
> >>>>>>>> but
> >>>>>>>>> there seems to be a consistent truncation of the date field just
> as
> >>>> you
> >>>>>>>> saw
> >>>>>>>>> in the app-name tag field.
> >>>>>>>>>
> >>>>>>>>> I appreciate your ongoing feedback and help. This issue is new to
> >> me
> >>>>>> and
> >>>>>>>>> will need a creative solution if the vendor has no answers.
> >>>>>>>>>
> >>>>>>>>> Many thanks,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> *Scott Slattery*
> >>>>>>>>>
> >>>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>>
> >>>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>>
> >>>>>>>>> motorolasolutions.com
> >>>>>>>>>
> >>>>>>>>> *O: 602.529.8226*
> >>>>>>>>>
> >>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, Mar 24, 2021 at 2:14 PM David Lang <david@lang.hm>
> wrote:
> >>>>>>>>>
> >>>>>>>>>> when you say 'each is incorrect but the same format'
> >>>>>>>>>>
> >>>>>>>>>> does that mean that every log has the year missing? or that
> every
> >>>> log
> >>>>>>>> is
> >>>>>>>>>> combining the logs together?
> >>>>>>>>>>
> >>>>>>>>>> I'll note that it's possible to define a custom parser using
> the
> >>>>>>>>>> mmnormalize
> >>>>>>>>>> library and add it to the parsing stack. I helped define the
> need
> >>>> for
> >>>>>>>> such
> >>>>>>>>>> a
> >>>>>>>>>> capability, but haven't used it yet.
> >>>>>>>>>>
> >>>>>>>>>> David Lang
> >>>>>>>>>>
> >>>>>>>>>> On Wed, 24 Mar 2021, Scott Slattery wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Date: Wed, 24 Mar 2021 08:58:07 -0700
> >>>>>>>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
> >>>>>>>>>>> To: mariusz.kruk@safecomp.com
> >>>>>>>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
> >>>>>>>> david@lang.hm
> >>>>>>>>>>>
> >>>>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>>>>>>>>>
> >>>>>>>>>>> Yes, all the logs are coming in this way. There are three
> >> separate
> >>>>>>>> logs
> >>>>>>>>>>> being sent and each is incorrect in its own right but the same
> >>>> format
> >>>>>>>>>>> behavior is consistent across each of the three logs. I will
> look
> >>>>>>>> further
> >>>>>>>>>>> into the suggestions you have provided and have already reached
> >> out
> >>>>>>>> to
> >>>>>>>>>> the
> >>>>>>>>>>> appliance support team and will raise the nonconformance with
> the
> >>>>>>>> RFC.
> >>>>>>>>>>>
> >>>>>>>>>>> This does provide me some valuable guidance and I thank each of
> >> you
> >>>>>>>> for
> >>>>>>>>>>> your feedback.
> >>>>>>>>>>>
> >>>>>>>>>>> *Scott Slattery*
> >>>>>>>>>>>
> >>>>>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>>>>
> >>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>>>>
> >>>>>>>>>>> motorolasolutions.com
> >>>>>>>>>>>
> >>>>>>>>>>> *O: 602.529.8226*
> >>>>>>>>>>>
> >>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com>
> >>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> There's something strange with this log entry. Notice that it
> >>>> looks
> >>>>>>>> as
> >>>>>>>>>> if
> >>>>>>>>>>>> it were more than one log entries joined with LF character
> >>>> (escaped
> >>>>>>>> on
> >>>>>>>>>>>> input as #012). And all "further" entries start with a - still
> >>>>>>>>>>>> nonconformant to the RFC but "fuller" - timestamp containing
> >>>>>>>> year.It's
> >>>>>>>>>> the
> >>>>>>>>>>>> leading event that seems to be trimmed in the front somehow.
> Do
> >>>> all
> >>>>>>>>>> events
> >>>>>>>>>>>> arrive like this?Wys?ano z telefonu Samsung<div>
> >>>>>>>>>>>> </div><div>
> >>>>>>>>>>>> </div><div>-------- Original message
> >>>>>>>>>>>> --------</div><div>From: Scott Slattery via rsyslog <
> >>>>>>>>>>>> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021 01:52
> >>>>>>>>>>>> (GMT+01:00) </div><div>To: David Lang <david@lang.hm>
> >>>>>>>> </div><div>Cc:
> >>>>>>>>>>>> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott
> >>>>>>>> Slattery
> >>>>>>>>>> via
> >>>>>>>>>>>> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re:
> >>>>>>>> [rsyslog]
> >>>>>>>>>>>> Altering forwarded logfile names </div><div>
> >>>>>>>>>>>> </div>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi David, fortunately I had already done this. I'm including
> an
> >>>>>>>> actual
> >>>>>>>>>> log
> >>>>>>>>>>>> entry but have anonymized the data to keep the actual user and
> >>>> email
> >>>>>>>>>>>> address confidential:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Debug line with all properties:
> >>>>>>>>>>>> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
> >>>>>>>>>>>> , fromhost-ip: '10.41.102.143'
> >>>>>>>>>>>> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
> >>>>>>>>>>>> , PRI: 13,
> >>>>>>>>>>>> syslogtag '03-23T16:'
> >>>>>>>>>>>> , programname: '03-23T16'
> >>>>>>>>>>>> , APP-NAME: '03-23T16'
> >>>>>>>>>>>> , PROCID: '-'
> >>>>>>>>>>>> , MSGID: '-'
> >>>>>>>>>>>> , TIMNESTAMP: 'Mar 23 21:47:13'
> >>>>>>>>>>>> , STRUCTURED-DATA: '-'
> >>>>>>>>>>>> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
> >>>>>>>>>> ACCESS_GATEWAY
> >>>>>>>>>>>> ACCESS AUTHZ SESSION INFO USER_SESSION
> >>>>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>> SUBJECT="
> >>>>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >>>>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>>>>>> apigniodashboard-uat.mycompany.com"
> >>>>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> >>>>>>>>>> REMOTE_IP="10.44.65.38"
> >>>>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >>>>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>>>>>> Safari/537.36"]
> >>>>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >>>>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>> POLICY
> >>>>>>>>>> INFO
> >>>>>>>>>>>> USER_AUTHZ
> >>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
> >>>> RESOURCE="/_dash-dependencies"
> >>>>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED"
> DURATION="0"
> >>>>>>>>>> APP="Ignio
> >>>>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW"
> REASON="N/A
> >> -
> >>>>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
> >>>> oag_username=
> >>>>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
> >>>>>>>>>> apigniodashboard-uat.mycompany.com
> >>>>>>>>>>>> UserName=Joe.User@mycompantions.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >>>>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
> >> Mac
> >>>>>>>> OS X
> >>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> >>>> Chrome/88.0.4324.192
> >>>>>>>>>>>> Safari/537.36 creationTime=1616536016811
> >>>> maxInactiveInterval=3600000
> >>>>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >>>>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
> >> Intel
> >>>>>>>> Mac
> >>>>>>>>>> OS X
> >>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> >>>> Chrome/88.0.4324.192
> >>>>>>>>>>>> Safari/537.36"] allow access to
> >>>>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
> >>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>>>>>> SESSION
> >>>>>>>>>> INFO
> >>>>>>>>>>>> USER_SESSION
> >>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>>>>>> SUBJECT="Joe.User@mycompan'
> >>>>>>>>>>>> *escaped msg*: '47:20.708-05:00
> >> AUSE1OAGATST02.aws.mycompany.com
> >>>>>>>>>>>> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
> >>>>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>> SUBJECT="
> >>>>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >>>>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>>>>>> apigniodashboard-uat.mycompany.com"
> >>>>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> >>>>>>>>>> REMOTE_IP="10.44.65.38"
> >>>>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >>>>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>>>>>> Safari/537.36"]
> >>>>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >>>>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>> POLICY
> >>>>>>>>>> INFO
> >>>>>>>>>>>> USER_AUTHZ
> >>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
> >>>> RESOURCE="/_dash-dependencies"
> >>>>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED"
> DURATION="0"
> >>>>>>>>>> APP="Ignio
> >>>>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW"
> REASON="N/A
> >> -
> >>>>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
> >>>> oag_username=
> >>>>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
> >>>>>>>>>> apigniodashboard-uat.mycompany.com
> >>>>>>>>>>>> UserName=Joe.User@mycompantions.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >>>>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
> >> Mac
> >>>>>>>> OS X
> >>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> >>>> Chrome/88.0.4324.192
> >>>>>>>>>>>> Safari/537.36 creationTime=1616536016811
> >>>> maxInactiveInterval=3600000
> >>>>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >>>>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
> >> Intel
> >>>>>>>> Mac
> >>>>>>>>>> OS X
> >>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> >>>> Chrome/88.0.4324.192
> >>>>>>>>>>>> Safari/537.36"] allow access to
> >>>>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
> >>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>>>>>> SESSION
> >>>>>>>>>> INFO
> >>>>>>>>>>>> USER_SESSION
> >>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>>>>>> SUBJECT="Joe.User@mycompan'
> >>>>>>>>>>>> *rawmsg*: '03-23T16:47:20.708-05:00
> >>>>>>>> AUSE1OAGATST02.aws.mycompany.com
> >>>>>>>>>>>> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
> >>>>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>> SUBJECT="
> >>>>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
> >>>>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>>>>>> apigniodashboard-uat.mycompany.com"
> >>>>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
> >>>>>>>>>> REMOTE_IP="10.44.65.38"
> >>>>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
> >>>>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
> >>>>>>>>>> Safari/537.36"]
> >>>>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
> >>>>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
> >>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>> POLICY
> >>>>>>>>>> INFO
> >>>>>>>>>>>> USER_AUTHZ
> >>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
> >>>> RESOURCE="/_dash-dependencies"
> >>>>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED"
> DURATION="0"
> >>>>>>>>>> APP="Ignio
> >>>>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
> >>>>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW"
> REASON="N/A
> >> -
> >>>>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
> >>>> oag_username=
> >>>>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
> >>>>>>>>>> apigniodashboard-uat.mycompany.com
> >>>>>>>>>>>> UserName=Joe.User@mycompantions.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
> >>>>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
> >> Mac
> >>>>>>>> OS X
> >>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> >>>> Chrome/88.0.4324.192
> >>>>>>>>>>>> Safari/537.36 creationTime=1616536016811
> >>>> maxInactiveInterval=3600000
> >>>>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
> >>>>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
> >> Intel
> >>>>>>>> Mac
> >>>>>>>>>> OS X
> >>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
> >>>> Chrome/88.0.4324.192
> >>>>>>>>>>>> Safari/537.36"] allow access to
> >>>>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
> >>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
> >>>>>>>> SESSION
> >>>>>>>>>> INFO
> >>>>>>>>>>>> USER_SESSION
> >>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
> >>>>>>>>>>>> SUBJECT="Joe.User@mycompan'
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> By inspecting the rawmsg, I can see that field four
> >>>>>>>> (space-delimited)
> >>>>>>>>>>>> indicates this is the ACCESS log. So if I were able to extract
> >> the
> >>>>>>>> log
> >>>>>>>>>>>> identifier from the msg, I could then write all access logs to
> >> the
> >>>>>>>> same
> >>>>>>>>>>>> daily file. There are other formats as well from the same
> device
> >>>>>>>> but the
> >>>>>>>>>>>> idea is the same.
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Scott Slattery*
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>>>>>
> >>>>>>>>>>>> motorolasolutions.com
> >>>>>>>>>>>>
> >>>>>>>>>>>> *O: 602.529.8226*
> >>>>>>>>>>>>
> >>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm>
> >> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> the source logfile name is not included in the payload by the
> >>>>>>>> syslog
> >>>>>>>>>>>> spec.
> >>>>>>>>>>>>> It
> >>>>>>>>>>>>> may be in the case of your appliance, but we would need to
> see
> >> a
> >>>>>>>> sample
> >>>>>>>>>>>>> log to
> >>>>>>>>>>>>> understand ho to parse it.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> based on your template, you are using app-name, which may be
> >>>> listed
> >>>>>>>>>>>>> separtely if
> >>>>>>>>>>>>> it's a RFC5424 format log, or may be part of the syslog tag
> if
> >>>>>>>> it's a
> >>>>>>>>>>>>> RFC3164
> >>>>>>>>>>>>> format log over the wire (neither format has a way to
> specify a
> >>>>>>>> source
> >>>>>>>>>>>> log
> >>>>>>>>>>>>> file
> >>>>>>>>>>>>> by default)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> you can look at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
> >>>>>>>>>>>>> and see the *-cc
> >>>>>>>>>>>>> options that you could apply to the app-name to eliminate
> >> control
> >>>>>>>>>>>>> characters.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Again, we really need to see the original log message to
> >>>> understand
> >>>>>>>>>>>> what's
> >>>>>>>>>>>>> what.
> >>>>>>>>>>>>> Please log it with the templateRSYSLOG_DebugFormat so we can
> >> see
> >>>>>>>>>> exactly
> >>>>>>>>>>>>> what is
> >>>>>>>>>>>>> sent over the wire and how rsyslog has parsed it.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> David Lang
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
> >>>>>>>>>>>>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com
> >
> >>>>>>>>>>>>>> To: John Chivian <jchivian@chivian.com>
> >>>>>>>>>>>>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
> >>>>>>>>>>>>>> rsyslog-users <rsyslog@lists.adiscon.com>
> >>>>>>>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks, John, let me try to clarify what I mean.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Normally when I forward from a remote server to the central
> >> log
> >>>>>>>>>>>> server, I
> >>>>>>>>>>>>>> can include a tag that can then be used to determine the
> file
> >>>>>>>> name I
> >>>>>>>>>>>> want
> >>>>>>>>>>>>>> on the central server. Since I have no real way to include
> >> this
> >>>>>>>> tag
> >>>>>>>>>>>> from
> >>>>>>>>>>>>>> the appliance, this is not an option.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'm looking for a way of inspecting the incoming packets to
> >>>>>>>>>> determining
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>> source logfile name (which is included in the payload) and
> use
> >>>>>>>> that
> >>>>>>>>>>>>>> filename on the target central server. Since there are
> >> multiple
> >>>>>>>> logs
> >>>>>>>>>>>>> being
> >>>>>>>>>>>>>> sent (access, audit, monitor, etc.), I'd like to segregate
> >> these
> >>>>>>>> into
> >>>>>>>>>>>>> their
> >>>>>>>>>>>>>> own files. I'm already using a template with the host
> >>>> information
> >>>>>>>> to
> >>>>>>>>>>>>>> dynamically create the file names. I just don't know how I
> can
> >>>> go
> >>>>>>>>>>>> beyond
> >>>>>>>>>>>>>> this to also include the source logname.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Here's the template I'm using. It works for all other hosts
> >>>> where
> >>>>>>>> I
> >>>>>>>>>> can
> >>>>>>>>>>>>>> configure the tag but I get garbage names from the
> appliance.
> >> I
> >>>>>>>> had
> >>>>>>>>>>>> hoped
> >>>>>>>>>>>>>> that the appliance included some standard syslog tags but it
> >>>>>>>> doesn't
> >>>>>>>>>>>> seem
> >>>>>>>>>>>>>> so.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> template(name="DynRemoteLogFile" type="string"
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *Scott Slattery*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> motorolasolutions.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *O: 602.529.8226*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <
> >>>>>>>> jchivian@chivian.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Your use of the term “file name” is confusing. When
> senders
> >>>>>>>> deliver
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>> rsyslog over the network there is no exchange of files or
> >>>>>>>> filenames,
> >>>>>>>>>>>>> only
> >>>>>>>>>>>>>>> packets of information. Those packets are expected to be
> in
> >> a
> >>>>>>>> format
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>>> syslog understands such that useful information (header
> >>>> elements
> >>>>>>>> and
> >>>>>>>>>>>>>>> message body) may be parsed from them. If you as the
> rsyslog
> >>>>>>>> admin
> >>>>>>>>>>>>> choose
> >>>>>>>>>>>>>>> to use some of that header information to compose filenames
> >> for
> >>>>>>>>>> output
> >>>>>>>>>>>>>>> files, then yes you are sort of at the mercy of the senders
> >>>>>>>> content
> >>>>>>>>>>>>>>> (especially if the sender doesn’t follow the syslog rules).
> >>>>>>>> However,
> >>>>>>>>>>>>> there
> >>>>>>>>>>>>>>> are functions in the advanced syntax that can be used to
> >>>> perform
> >>>>>>>> the
> >>>>>>>>>>>>> type
> >>>>>>>>>>>>>>> of character replacements you’re talking about.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> It is common practice to use the syslog header/rsyslog
> >> property
> >>>>>>>>>>>> element
> >>>>>>>>>>>>>>> called “hostname” for just such purposes. Is this what
> >> you’re
> >>>>>>>>>> talking
> >>>>>>>>>>>>>>> about? You’d have to provide your configuration for real
> >>>>>>>> analysis,
> >>>>>>>>>> at
> >>>>>>>>>>>>>>> least the part you perceive to be responsible for the
> >> problem.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
> >>>>>>>>>>>>>>> rsyslog@lists.adiscon.com> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I have a configured central log collector using rsyslog. A
> >> few
> >>>>>>>> of
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> devices forwarding their logs are appliances that have no
> >>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>>> options other than the IP forwarding address and
> protocol. I
> >>>>>>>> cannot
> >>>>>>>>>>>>>>> control
> >>>>>>>>>>>>>>>> what file names are being sent.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Unfortunately, they are sending unintelligible file names
> >> with
> >>>>>>>>>>>>> characters
> >>>>>>>>>>>>>>>> that normally would be escaped. Is there any way I can
> >> control
> >>>>>>>> or
> >>>>>>>>>>>>> alter
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> incoming file name to normalize it to avoid these odd
> >>>>>>>> characters?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> For example, could I establish a character map that maps
> the
> >>>>>>>>>>>> unallowed
> >>>>>>>>>>>>>>>> character to something acceptable?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> thanks,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> *Scott Slattery*
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> *Sr. Systems & Cloud Architect*
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> motorolasolutions.com
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> *O: 602.529.822*
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> *For more information on how and why we collect your
> >> personal
> >>>>>>>>>>>>>>>> information, please visit our Privacy Policy
> >>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>>>>>>>>>>> .*
> >>>>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>>>> rsyslog mailing list
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
> >>>>>>>>>>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
> >>>>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
> ARCHIVED
> >>>> by
> >>>>>>>> a
> >>>>>>>>>>>>> myriad
> >>>>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> >> POST
> >>>>>>>> if
> >>>>>>>>>> you
> >>>>>>>>>>>>>>> DON'T LIKE THAT.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *For more information on how and why we collect your
> personal
> >>>>>>>>>>>>>> information, please visit our Privacy Policy
> >>>>>>>>>>>>>> <
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>>>>>>>>> .*
> >>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>> rsyslog mailing list
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
> >>>>>>>>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
> >>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> >> by
> >>>> a
> >>>>>>>>>>>> myriad
> >>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> POST
> >>>> if
> >>>>>>>> you
> >>>>>>>>>>>>> DON'T LIKE THAT.
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> *For more information on how and why we collect your personal
> >>>>>>>>>>>> information, please visit our Privacy Policy
> >>>>>>>>>>>> <
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>>>>>>>> .*
> >>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>> rsyslog mailing list
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
> >>>>>>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
> >>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> by
> >> a
> >>>>>>>> myriad
> >>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> POST
> >> if
> >>>>>>>> you
> >>>>>>>>>>>> DON'T LIKE THAT.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> *For more information on how and why we collect your personal
> >>>>>>>>> information, please visit our Privacy Policy
> >>>>>>>>> <
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >>>>>>>>> .*
> >>>>>>>>> _______________________________________________
> >>>>>>>>> rsyslog mailing list
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=7hc9b2gSfvsfQzXvsRVXqGeP4Rz_tJtrykpjqkGBIdk&e=
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=bDybhvts9KqI-WEfMHo7O40ulQK5M6Tg4tnKNRRIhE8&e=
> >>>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=zPhpcUPlK_CWLiU75_WwjyQGz5mFTiXc1yyBIfppkrI&e=
> >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>>>>> myriad
> >>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> >> you
> >>>>>>>> DON'T LIKE THAT.
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >
> >

--

*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Altering forwarded logfile names [ In reply to ]

rsyslog at lists

Mar 26, 2021, 12:55 AM

Post #28 of 30 (545 views)

Permalink

I think I guess what is going on. Probably a message starts, without pri,
with a number. If so, that triggers octet-counted framing, because that's
what it really indicates. So the number becomes the message length.

Try param

supportOctedtCountedFraming="off"

Doc:

https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html

Rainer

El jue, 25 mar 2021 a las 23:37, Scott Slattery (<
scott.slattery@motorolasolutions.com>) escribió:

> Upon further review, I'm seeing packet lengths from 4434 down to 111. It
> appears once the packet length exceeds 2919 the trailing 0x0a is missing
> from any remaining packets larger than 2919.
>
> If I sort by packet size, it jumps from 2919 to 2974 then all further line
> feed character is missing.
>
> *Scott Slattery*
>
> *Sr. Systems & Cloud Architect*
>
> *Cloud, Compute, Information & Architecture Team*
>
> motorolasolutions.com
>
> *O: 602.529.8226*
>
> *E*: Scott.Slattery@MotorolaSolutions.com
>
>
>
>
> On Thu, Mar 25, 2021 at 3:24 PM Scott Slattery <
> scott.slattery@motorolasolutions.com> wrote:
>
>>
>> *Rsyslogd version:*
>> rsyslogd 8.24.0-52.amzn2, compiled with:
>> PLATFORM: x86_64-koji-linux-gnu
>> PLATFORM (lsb_release -d):
>> FEATURE_REGEXP: Yes
>> GSSAPI Kerberos 5 support: Yes
>> FEATURE_DEBUG (debug build, slow code): No
>> 32bit Atomic operations supported: Yes
>> 64bit Atomic operations supported: Yes
>> memory allocator: system default
>> Runtime Instrumentation (slow code): No
>> uuid support: Yes
>> Number of Bits in RainerScript integers: 64
>>
>> Here is my config:
>> $MaxMessageSize 64k
>> $EscapeControlCharactersOnReceive on
>>
>> module(load="imuxsock")
>> module(load="imjournal" StateFile="imjournal.state")
>> module(load="imklog")
>> module(load="immark")
>> module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat"
>> DirCreateMode="0705" FileCreateMode="0704")
>> module(load="impstats" interval="600" log.syslog="off" severity="7"
>> log.file="/var/log/impstats.log")
>>
>> $DynaFileCacheSize 100
>> global(workDirectory="/var/lib/rsyslog" localHostname="cloudlogmaster01")
>>
>> $MainMsgQueueSize 200000
>> $MainMsgQueueHighWaterMark 120000
>> $MainMsgQueueDiscardMark 160000
>> $MainMsgQueueWorkerThreads 100
>> $RulesetCreateMainQueue on
>>
>> $imjournalRatelimitInterval 0
>> $imjournalRatelimitBurst 0
>> #### RULES ####
>>
>> template(name="debug" type="string" string="Debug line with all
>> properties:\nFROMHOST: '%FROMHOST%'\n, FROMHOST-IP: '%fromhost-ip%'\n,
>> HOSTNAME: '%HOSTNAME%'\n, PRI: %PRI%,\nSYSLOGTAG '%syslogtag%'\n,
>> PROGRAMNAME: '%programname%'\n, APP-NAME: '%app-name%'\n, PROCID:
>> '%PROCID%'\n, MSGID: '%MSGID%'\n, TIMESTAMP: '%TIMESTAMP%'\n,
>> STRUCTURED-DATA: '%STRUCTURED-DATA%'\n, msg: '%msg%'\nRAWMSG:
>> '%rawmsg%',\n\nRAWMSG-AFTER-PRI: '%rawmsg-after-pri%'\nJSONMESG:
>> '%jsonmesg%'\n")
>>
>> template(name="DynRemoteLogFile" type="string"
>> string="/splunklog/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
>>
>> template(name="MalformedMsgFormatter" type="string"
>> string="%timegenerated% %fromhost% %rawmsg:::drop-last-lf%\n")
>> template(name="DynRemoteMalformed" type="string"
>> string="/splunklog/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-oag-audit.log")
>>
>> template(name="DynRemoteLogFile" type="list") {
>> constant(value="/splunklog/remote/")
>> property(name="FROMHOST")
>> constant(value="-")
>> property(name="FROMHOST-IP")
>> constant(value="/")
>> property(name="$YEAR")
>> constant(value="-")
>> property(name="$MONTH")
>> constant(value="-")
>> property(name="$DAY")
>> constant(value="-")
>> property(name="syslogtag" position.from="1" position.to="32")
>> constant(value="-")
>> property(name="app-name" controlcharacters="drop")
>> constant(value=".log")
>> }
>>
>> ruleset(name="RemoteServer"){
>> if ($fromhost-ip startswith '10.41.102') then {
>> action(type="omfile" dynafile="DynRemoteMalformed"
>> template="MalformedMsgFormatter")
>> stop
>> }
>>
>> action(type="omfile" dynaFile="DynRemoteLogFile")
>> }
>>
>>
>> # module imtcp
>> module(load="imtcp" KeepAlive="on")
>> input(type="imtcp" port="10514" ruleset="RemoteServer")
>>
>>
>> #kern.* /dev/console
>> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>> authpriv.* /var/log/secure
>> mail.* /var/log/maillog
>> cron.* /var/log/cron
>> *.emerg :omusrmsg:*
>> uucp,news.crit /var/log/spooler
>> local7.* /var/log/boot.log
>>
>> *Scott Slattery*
>>
>> *Sr. Systems & Cloud Architect*
>>
>> *Cloud, Compute, Information & Architecture Team*
>>
>> motorolasolutions.com
>>
>> *O: 602.529.8226*
>>
>> *E*: Scott.Slattery@MotorolaSolutions.com
>>
>>
>>
>>
>> On Thu, Mar 25, 2021 at 3:20 PM David Lang <david@lang.hm> wrote:
>>
>>> what version of rsyslog are you running. can you post your full config?
>>>
>>> if you are receiving via TCP and it's not splitting the logs based on
>>> newlines,
>>> something very odd is happening.
>>>
>>> David Lang
>>>
>>> On Thu, 25 Mar 2021, Scott Slattery wrote:
>>>
>>> > Date: Thu, 25 Mar 2021 15:07:22 -0700
>>> > From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>> > To: David Lang <david@lang.hm>
>>> > Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
>>> > rsyslog-users <rsyslog@lists.adiscon.com>,
>>> mariusz.kruk@safecomp.com
>>> > Subject: Re: [rsyslog] Altering forwarded logfile names
>>> >
>>> > So I'm looking at the pcap and on the first log record the header
>>> > information looks fine and the record terminated by 0x0A line feed.
>>> >
>>> > When I look at the record containing this record written to the rsyslog
>>> > file, the record length is 65586 (64k) long which is what I think you
>>> were
>>> > referring to earlier. Why would rsyslog write such a long record? I
>>> > initially thought my data may be getting truncated so I updated
>>> > '$MaxMessageSize 64k'. Any suggestion how I fix this behavior.
>>> >
>>> > I can confirm that each log record in the pcap is terminated by 0x0A
>>> and
>>> > looks in tact so must be something I need to do on rsyslog to fix this.
>>> >
>>> > *Scott Slattery*
>>> >
>>> > *Sr. Systems & Cloud Architect*
>>> >
>>> > *Cloud, Compute, Information & Architecture Team*
>>> >
>>> > motorolasolutions.com
>>> >
>>> > *O: 602.529.8226*
>>> >
>>> > *E*: Scott.Slattery@MotorolaSolutions.com
>>> >
>>> >
>>> >
>>> >
>>> > On Thu, Mar 25, 2021 at 1:15 PM David Lang <david@lang.hm> wrote:
>>> >
>>> >> you may want to capture with -X so that it decodes it into hex and
>>> you can
>>> >> see
>>> >> newlines vs linefeeds
>>> >>
>>> >> David Lang
>>> >>
>>> >> On Thu, 25 Mar 2021, Scott Slattery wrote:
>>> >>
>>> >>> Date: Thu, 25 Mar 2021 12:47:43 -0700
>>> >>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>> >>> To: David Lang <david@lang.hm>
>>> >>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
>>> >>> rsyslog-users <rsyslog@lists.adiscon.com>,
>>> mariusz.kruk@safecomp.com
>>> >>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>> >>>
>>> >>> Thanks David, I just grabbed my first capture and am looking at it.
>>> I'm
>>> >>> using TCP vs UDP but your comment makes sense. I'll certainly share
>>> with
>>> >>> you my observations once I have some data to refer to. This has been
>>> a
>>> >>> learning experience. It's also the first time I've ever seen such an
>>> >> issue
>>> >>> with rsyslog so it's a bit of a puzzle.
>>> >>>
>>> >>> *Scott Slattery*
>>> >>>
>>> >>> *Sr. Systems & Cloud Architect*
>>> >>>
>>> >>> *Cloud, Compute, Information & Architecture Team*
>>> >>>
>>> >>> motorolasolutions.com
>>> >>>
>>> >>> *O: 602.529.8226*
>>> >>>
>>> >>> *E*: Scott.Slattery@MotorolaSolutions.com
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Thu, Mar 25, 2021 at 12:32 PM David Lang <david@lang.hm> wrote:
>>> >>>
>>> >>>> the rawmsg field in the debugformat output shows exactly what
>>> rsyslog is
>>> >>>> seeing.
>>> >>>>
>>> >>>> the reason I asked you to check multiple entries is that if rsyslog
>>> does
>>> >>>> not see
>>> >>>> the separator (due to either multiple messages in one UDP packet, or
>>> >>>> missing
>>> >>>> newlines in a TCP stream) it will combine what are intended as
>>> multiple
>>> >>>> messages
>>> >>>> together into one message as far as rsyslog is concerned, and only
>>> break
>>> >>>> it into
>>> >>>> multiple messages when it hits the configured maxmessagesize size.
>>> That
>>> >>>> breaks
>>> >>>> things at arbitrary points in the message, so as you look at
>>> multiple
>>> >>>> messages,
>>> >>>> they will not all start the same way.
>>> >>>>
>>> >>>> a tcpdump -s0 -A -n (with the appropriate port/ip filter) will also
>>> show
>>> >>>> exactly
>>> >>>> what's happening over the wire.
>>> >>>>
>>> >>>> David Lang
>>> >>>>
>>> >>>> On Thu, 25 Mar 2021, Scott Slattery wrote:
>>> >>>>
>>> >>>>> Date: Thu, 25 Mar 2021 12:23:03 -0700
>>> >>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>> >>>>> To: Rainer Gerhards <rgerhards@hq.adiscon.com>
>>> >>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
>>> >> david@lang.hm
>>> >>>>> ,
>>> >>>>> mariusz.kruk@safecomp.com
>>> >>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>> >>>>>
>>> >>>>> Hey Guys, before I consume more of your time, I'm capturing the
>>> packets
>>> >>>>> from the source device to determine if any of the payload is lost
>>> on
>>> >> its
>>> >>>>> way to the rsyslog collector. After I analyze further, I'll share
>>> my
>>> >>>>> observations. I'm hoping nothing is getting dropped over the wire.
>>> >> Thanks
>>> >>>>> for your patience and help so far. I expect to have additional
>>> details
>>> >>>>> shortly.
>>> >>>>>
>>> >>>>> *Scott Slattery*
>>> >>>>>
>>> >>>>> *Sr. Systems & Cloud Architect*
>>> >>>>>
>>> >>>>> *Cloud, Compute, Information & Architecture Team*
>>> >>>>>
>>> >>>>> motorolasolutions.com
>>> >>>>>
>>> >>>>> *O: 602.529.8226*
>>> >>>>>
>>> >>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Thu, Mar 25, 2021 at 12:51 AM Rainer Gerhards <
>>> >>>> rgerhards@hq.adiscon.com>
>>> >>>>> wrote:
>>> >>>>>
>>> >>>>>> Sorry if I didn't catch it, but did you post the configuration?
>>> If
>>> >>>>>> not, it would probably be useful.
>>> >>>>>>
>>> >>>>>> Looking at the message properties, this looks like the message was
>>> >>>>>> received via UDP but had a LF inside it which was intended as a
>>> frame
>>> >>>>>> separator. If so, that is another bug in the vendor's
>>> implementation.
>>> >>>>>> Framing is used for TCP, but not UDP. For UDP it is strictly one
>>> >>>>>> message per package.
>>> >>>>>>
>>> >>>>>> HTH,
>>> >>>>>> Rainer
>>> >>>>>>
>>> >>>>>> El mié, 24 mar 2021 a las 22:23, Scott Slattery via rsyslog
>>> >>>>>> (<rsyslog@lists.adiscon.com>) escribió:
>>> >>>>>>>
>>> >>>>>>> From what I can discern since there are lots of transactions,
>>> there
>>> >>>> seem
>>> >>>>>> to
>>> >>>>>>> be multiple payload formats. For all that I have so far
>>> analyzed, the
>>> >>>>>> date
>>> >>>>>>> truncation issue is the same regardless of the payload received.
>>> Like
>>> >>>>>> you,
>>> >>>>>>> I've never used mmnormalize but I will look at it as an option.
>>> I've
>>> >>>>>>> reached out to the product support team to better understand if
>>> there
>>> >>>> is
>>> >>>>>> a
>>> >>>>>>> appliance-side misconfiguration and something that can be
>>> corrected
>>> >>>> there
>>> >>>>>>> or if I need to make adjustments on the rsyslog side to clean up
>>> the
>>> >>>>>>> records.
>>> >>>>>>>
>>> >>>>>>> What I meant is that there are multiple message formats (payload
>>> >> data)
>>> >>>>>> but
>>> >>>>>>> there seems to be a consistent truncation of the date field just
>>> as
>>> >> you
>>> >>>>>> saw
>>> >>>>>>> in the app-name tag field.
>>> >>>>>>>
>>> >>>>>>> I appreciate your ongoing feedback and help. This issue is new
>>> to me
>>> >>>> and
>>> >>>>>>> will need a creative solution if the vendor has no answers.
>>> >>>>>>>
>>> >>>>>>> Many thanks,
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> *Scott Slattery*
>>> >>>>>>>
>>> >>>>>>> *Sr. Systems & Cloud Architect*
>>> >>>>>>>
>>> >>>>>>> *Cloud, Compute, Information & Architecture Team*
>>> >>>>>>>
>>> >>>>>>> motorolasolutions.com
>>> >>>>>>>
>>> >>>>>>> *O: 602.529.8226*
>>> >>>>>>>
>>> >>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> On Wed, Mar 24, 2021 at 2:14 PM David Lang <david@lang.hm>
>>> wrote:
>>> >>>>>>>
>>> >>>>>>>> when you say 'each is incorrect but the same format'
>>> >>>>>>>>
>>> >>>>>>>> does that mean that every log has the year missing? or that
>>> every
>>> >> log
>>> >>>>>> is
>>> >>>>>>>> combining the logs together?
>>> >>>>>>>>
>>> >>>>>>>> I'll note that it's possible to define a custom parser using
>>> the
>>> >>>>>>>> mmnormalize
>>> >>>>>>>> library and add it to the parsing stack. I helped define the
>>> need
>>> >> for
>>> >>>>>> such
>>> >>>>>>>> a
>>> >>>>>>>> capability, but haven't used it yet.
>>> >>>>>>>>
>>> >>>>>>>> David Lang
>>> >>>>>>>>
>>> >>>>>>>> On Wed, 24 Mar 2021, Scott Slattery wrote:
>>> >>>>>>>>
>>> >>>>>>>>> Date: Wed, 24 Mar 2021 08:58:07 -0700
>>> >>>>>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>> >>>>>>>>> To: mariusz.kruk@safecomp.com
>>> >>>>>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
>>> >>>>>> david@lang.hm
>>> >>>>>>>>>
>>> >>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>> >>>>>>>>>
>>> >>>>>>>>> Yes, all the logs are coming in this way. There are three
>>> separate
>>> >>>>>> logs
>>> >>>>>>>>> being sent and each is incorrect in its own right but the same
>>> >> format
>>> >>>>>>>>> behavior is consistent across each of the three logs. I will
>>> look
>>> >>>>>> further
>>> >>>>>>>>> into the suggestions you have provided and have already
>>> reached out
>>> >>>>>> to
>>> >>>>>>>> the
>>> >>>>>>>>> appliance support team and will raise the nonconformance with
>>> the
>>> >>>>>> RFC.
>>> >>>>>>>>>
>>> >>>>>>>>> This does provide me some valuable guidance and I thank each
>>> of you
>>> >>>>>> for
>>> >>>>>>>>> your feedback.
>>> >>>>>>>>>
>>> >>>>>>>>> *Scott Slattery*
>>> >>>>>>>>>
>>> >>>>>>>>> *Sr. Systems & Cloud Architect*
>>> >>>>>>>>>
>>> >>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>> >>>>>>>>>
>>> >>>>>>>>> motorolasolutions.com
>>> >>>>>>>>>
>>> >>>>>>>>> *O: 602.529.8226*
>>> >>>>>>>>>
>>> >>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com>
>>> >> wrote:
>>> >>>>>>>>>
>>> >>>>>>>>>> There's something strange with this log entry. Notice that it
>>> >> looks
>>> >>>>>> as
>>> >>>>>>>> if
>>> >>>>>>>>>> it were more than one log entries joined with LF character
>>> >> (escaped
>>> >>>>>> on
>>> >>>>>>>>>> input as #012). And all "further" entries start with a - still
>>> >>>>>>>>>> nonconformant to the RFC but "fuller" - timestamp containing
>>> >>>>>> year.It's
>>> >>>>>>>> the
>>> >>>>>>>>>> leading event that seems to be trimmed in the front somehow.
>>> Do
>>> >> all
>>> >>>>>>>> events
>>> >>>>>>>>>> arrive like this?Wys?ano z telefonu Samsung<div>
>>> >>>>>>>>>> </div><div>
>>> >>>>>>>>>> </div><div>-------- Original message
>>> >>>>>>>>>> --------</div><div>From: Scott Slattery via rsyslog <
>>> >>>>>>>>>> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021 01:52
>>> >>>>>>>>>> (GMT+01:00) </div><div>To: David Lang <david@lang.hm>
>>> >>>>>> </div><div>Cc:
>>> >>>>>>>>>> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott
>>> >>>>>> Slattery
>>> >>>>>>>> via
>>> >>>>>>>>>> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re:
>>> >>>>>> [rsyslog]
>>> >>>>>>>>>> Altering forwarded logfile names </div><div>
>>> >>>>>>>>>> </div>
>>> >>>>>>>>>>
>>> >>>>>>>>>> Hi David, fortunately I had already done this. I'm including
>>> an
>>> >>>>>> actual
>>> >>>>>>>> log
>>> >>>>>>>>>> entry but have anonymized the data to keep the actual user and
>>> >> email
>>> >>>>>>>>>> address confidential:
>>> >>>>>>>>>>
>>> >>>>>>>>>> Debug line with all properties:
>>> >>>>>>>>>> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
>>> >>>>>>>>>> , fromhost-ip: '10.41.102.143'
>>> >>>>>>>>>> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
>>> >>>>>>>>>> , PRI: 13,
>>> >>>>>>>>>> syslogtag '03-23T16:'
>>> >>>>>>>>>> , programname: '03-23T16'
>>> >>>>>>>>>> , APP-NAME: '03-23T16'
>>> >>>>>>>>>> , PROCID: '-'
>>> >>>>>>>>>> , MSGID: '-'
>>> >>>>>>>>>> , TIMNESTAMP: 'Mar 23 21:47:13'
>>> >>>>>>>>>> , STRUCTURED-DATA: '-'
>>> >>>>>>>>>> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
>>> >>>>>>>> ACCESS_GATEWAY
>>> >>>>>>>>>> ACCESS AUTHZ SESSION INFO USER_SESSION
>>> >>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>> >> SUBJECT="
>>> >>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>> >>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>> >>>>>>>>>> apigniodashboard-uat.mycompany.com"
>>> >>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>> >>>>>>>> REMOTE_IP="10.44.65.38"
>>> >>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>> >>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>> >>>>>>>> Safari/537.36"]
>>> >>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>> >>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>> >> POLICY
>>> >>>>>>>> INFO
>>> >>>>>>>>>> USER_AUTHZ
>>> >> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>> >>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
>>> >> RESOURCE="/_dash-dependencies"
>>> >>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED"
>>> DURATION="0"
>>> >>>>>>>> APP="Ignio
>>> >>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>> >>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW"
>>> REASON="N/A -
>>> >>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
>>> >> oag_username=
>>> >>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>>> >>>>>>>> apigniodashboard-uat.mycompany.com
>>> >>>>>>>>>> UserName=Joe.User@mycompantions.com
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>> >>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
>>> Mac
>>> >>>>>> OS X
>>> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>> >> Chrome/88.0.4324.192
>>> >>>>>>>>>> Safari/537.36 creationTime=1616536016811
>>> >> maxInactiveInterval=3600000
>>> >>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>> >>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
>>> Intel
>>> >>>>>> Mac
>>> >>>>>>>> OS X
>>> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>> >> Chrome/88.0.4324.192
>>> >>>>>>>>>> Safari/537.36"] allow access to
>>> >>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>>> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>> >>>>>> SESSION
>>> >>>>>>>> INFO
>>> >>>>>>>>>> USER_SESSION
>>> >>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>> >>>>>>>>>> SUBJECT="Joe.User@mycompan'
>>> >>>>>>>>>> *escaped msg*: '47:20.708-05:00
>>> AUSE1OAGATST02.aws.mycompany.com
>>> >>>>>>>>>> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
>>> >>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>> >> SUBJECT="
>>> >>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>> >>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>> >>>>>>>>>> apigniodashboard-uat.mycompany.com"
>>> >>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>> >>>>>>>> REMOTE_IP="10.44.65.38"
>>> >>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>> >>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>> >>>>>>>> Safari/537.36"]
>>> >>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>> >>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>> >> POLICY
>>> >>>>>>>> INFO
>>> >>>>>>>>>> USER_AUTHZ
>>> >> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>> >>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
>>> >> RESOURCE="/_dash-dependencies"
>>> >>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED"
>>> DURATION="0"
>>> >>>>>>>> APP="Ignio
>>> >>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>> >>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW"
>>> REASON="N/A -
>>> >>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
>>> >> oag_username=
>>> >>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>>> >>>>>>>> apigniodashboard-uat.mycompany.com
>>> >>>>>>>>>> UserName=Joe.User@mycompantions.com
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>> >>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
>>> Mac
>>> >>>>>> OS X
>>> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>> >> Chrome/88.0.4324.192
>>> >>>>>>>>>> Safari/537.36 creationTime=1616536016811
>>> >> maxInactiveInterval=3600000
>>> >>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>> >>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
>>> Intel
>>> >>>>>> Mac
>>> >>>>>>>> OS X
>>> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>> >> Chrome/88.0.4324.192
>>> >>>>>>>>>> Safari/537.36"] allow access to
>>> >>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>>> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>> >>>>>> SESSION
>>> >>>>>>>> INFO
>>> >>>>>>>>>> USER_SESSION
>>> >>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>> >>>>>>>>>> SUBJECT="Joe.User@mycompan'
>>> >>>>>>>>>> *rawmsg*: '03-23T16:47:20.708-05:00
>>> >>>>>> AUSE1OAGATST02.aws.mycompany.com
>>> >>>>>>>>>> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
>>> >>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>> >> SUBJECT="
>>> >>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>> >>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>> >>>>>>>>>> apigniodashboard-uat.mycompany.com"
>>> >>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>> >>>>>>>> REMOTE_IP="10.44.65.38"
>>> >>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>> >>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>> >>>>>>>> Safari/537.36"]
>>> >>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>> >>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>> >> POLICY
>>> >>>>>>>> INFO
>>> >>>>>>>>>> USER_AUTHZ
>>> >> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>> >>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
>>> >> RESOURCE="/_dash-dependencies"
>>> >>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED"
>>> DURATION="0"
>>> >>>>>>>> APP="Ignio
>>> >>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>> >>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW"
>>> REASON="N/A -
>>> >>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
>>> >> oag_username=
>>> >>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>>> >>>>>>>> apigniodashboard-uat.mycompany.com
>>> >>>>>>>>>> UserName=Joe.User@mycompantions.com
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>> >>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
>>> Mac
>>> >>>>>> OS X
>>> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>> >> Chrome/88.0.4324.192
>>> >>>>>>>>>> Safari/537.36 creationTime=1616536016811
>>> >> maxInactiveInterval=3600000
>>> >>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>> >>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
>>> Intel
>>> >>>>>> Mac
>>> >>>>>>>> OS X
>>> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>> >> Chrome/88.0.4324.192
>>> >>>>>>>>>> Safari/537.36"] allow access to
>>> >>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>>> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>> >>>>>> SESSION
>>> >>>>>>>> INFO
>>> >>>>>>>>>> USER_SESSION
>>> >>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>> >>>>>>>>>> SUBJECT="Joe.User@mycompan'
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> By inspecting the rawmsg, I can see that field four
>>> >>>>>> (space-delimited)
>>> >>>>>>>>>> indicates this is the ACCESS log. So if I were able to
>>> extract the
>>> >>>>>> log
>>> >>>>>>>>>> identifier from the msg, I could then write all access logs
>>> to the
>>> >>>>>> same
>>> >>>>>>>>>> daily file. There are other formats as well from the same
>>> device
>>> >>>>>> but the
>>> >>>>>>>>>> idea is the same.
>>> >>>>>>>>>>
>>> >>>>>>>>>> *Scott Slattery*
>>> >>>>>>>>>>
>>> >>>>>>>>>> *Sr. Systems & Cloud Architect*
>>> >>>>>>>>>>
>>> >>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>> >>>>>>>>>>
>>> >>>>>>>>>> motorolasolutions.com
>>> >>>>>>>>>>
>>> >>>>>>>>>> *O: 602.529.8226*
>>> >>>>>>>>>>
>>> >>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm>
>>> wrote:
>>> >>>>>>>>>>
>>> >>>>>>>>>>> the source logfile name is not included in the payload by the
>>> >>>>>> syslog
>>> >>>>>>>>>> spec.
>>> >>>>>>>>>>> It
>>> >>>>>>>>>>> may be in the case of your appliance, but we would need to
>>> see a
>>> >>>>>> sample
>>> >>>>>>>>>>> log to
>>> >>>>>>>>>>> understand ho to parse it.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> based on your template, you are using app-name, which may be
>>> >> listed
>>> >>>>>>>>>>> separtely if
>>> >>>>>>>>>>> it's a RFC5424 format log, or may be part of the syslog tag
>>> if
>>> >>>>>> it's a
>>> >>>>>>>>>>> RFC3164
>>> >>>>>>>>>>> format log over the wire (neither format has a way to
>>> specify a
>>> >>>>>> source
>>> >>>>>>>>>> log
>>> >>>>>>>>>>> file
>>> >>>>>>>>>>> by default)
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> you can look at
>>> >>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
>>> >>>>>>>>>>> and see the *-cc
>>> >>>>>>>>>>> options that you could apply to the app-name to eliminate
>>> control
>>> >>>>>>>>>>> characters.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Again, we really need to see the original log message to
>>> >> understand
>>> >>>>>>>>>> what's
>>> >>>>>>>>>>> what.
>>> >>>>>>>>>>> Please log it with the templateRSYSLOG_DebugFormat so we can
>>> see
>>> >>>>>>>> exactly
>>> >>>>>>>>>>> what is
>>> >>>>>>>>>>> sent over the wire and how rsyslog has parsed it.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> David Lang
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
>>> >>>>>>>>>>> wrote:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
>>> >>>>>>>>>>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com
>>> >
>>> >>>>>>>>>>>> To: John Chivian <jchivian@chivian.com>
>>> >>>>>>>>>>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
>>> >>>>>>>>>>>> rsyslog-users <rsyslog@lists.adiscon.com>
>>> >>>>>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> Thanks, John, let me try to clarify what I mean.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> Normally when I forward from a remote server to the central
>>> log
>>> >>>>>>>>>> server, I
>>> >>>>>>>>>>>> can include a tag that can then be used to determine the
>>> file
>>> >>>>>> name I
>>> >>>>>>>>>> want
>>> >>>>>>>>>>>> on the central server. Since I have no real way to include
>>> this
>>> >>>>>> tag
>>> >>>>>>>>>> from
>>> >>>>>>>>>>>> the appliance, this is not an option.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> I'm looking for a way of inspecting the incoming packets to
>>> >>>>>>>> determining
>>> >>>>>>>>>>> the
>>> >>>>>>>>>>>> source logfile name (which is included in the payload) and
>>> use
>>> >>>>>> that
>>> >>>>>>>>>>>> filename on the target central server. Since there are
>>> multiple
>>> >>>>>> logs
>>> >>>>>>>>>>> being
>>> >>>>>>>>>>>> sent (access, audit, monitor, etc.), I'd like to segregate
>>> these
>>> >>>>>> into
>>> >>>>>>>>>>> their
>>> >>>>>>>>>>>> own files. I'm already using a template with the host
>>> >> information
>>> >>>>>> to
>>> >>>>>>>>>>>> dynamically create the file names. I just don't know how I
>>> can
>>> >> go
>>> >>>>>>>>>> beyond
>>> >>>>>>>>>>>> this to also include the source logname.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> Here's the template I'm using. It works for all other hosts
>>> >> where
>>> >>>>>> I
>>> >>>>>>>> can
>>> >>>>>>>>>>>> configure the tag but I get garbage names from the
>>> appliance. I
>>> >>>>>> had
>>> >>>>>>>>>> hoped
>>> >>>>>>>>>>>> that the appliance included some standard syslog tags but it
>>> >>>>>> doesn't
>>> >>>>>>>>>> seem
>>> >>>>>>>>>>>> so.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> template(name="DynRemoteLogFile" type="string"
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> *Scott Slattery*
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> *Sr. Systems & Cloud Architect*
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> motorolasolutions.com
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> *O: 602.529.8226*
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <
>>> >>>>>> jchivian@chivian.com>
>>> >>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>> Your use of the term “file name” is confusing. When
>>> senders
>>> >>>>>> deliver
>>> >>>>>>>>>> to
>>> >>>>>>>>>>>>> rsyslog over the network there is no exchange of files or
>>> >>>>>> filenames,
>>> >>>>>>>>>>> only
>>> >>>>>>>>>>>>> packets of information. Those packets are expected to be
>>> in a
>>> >>>>>> format
>>> >>>>>>>>>>> that
>>> >>>>>>>>>>>>> syslog understands such that useful information (header
>>> >> elements
>>> >>>>>> and
>>> >>>>>>>>>>>>> message body) may be parsed from them. If you as the
>>> rsyslog
>>> >>>>>> admin
>>> >>>>>>>>>>> choose
>>> >>>>>>>>>>>>> to use some of that header information to compose
>>> filenames for
>>> >>>>>>>> output
>>> >>>>>>>>>>>>> files, then yes you are sort of at the mercy of the senders
>>> >>>>>> content
>>> >>>>>>>>>>>>> (especially if the sender doesn’t follow the syslog rules).
>>> >>>>>> However,
>>> >>>>>>>>>>> there
>>> >>>>>>>>>>>>> are functions in the advanced syntax that can be used to
>>> >> perform
>>> >>>>>> the
>>> >>>>>>>>>>> type
>>> >>>>>>>>>>>>> of character replacements you’re talking about.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> It is common practice to use the syslog header/rsyslog
>>> property
>>> >>>>>>>>>> element
>>> >>>>>>>>>>>>> called “hostname” for just such purposes. Is this what
>>> you’re
>>> >>>>>>>> talking
>>> >>>>>>>>>>>>> about? You’d have to provide your configuration for real
>>> >>>>>> analysis,
>>> >>>>>>>> at
>>> >>>>>>>>>>>>> least the part you perceive to be responsible for the
>>> problem.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Regards,
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
>>> >>>>>>>>>>>>> rsyslog@lists.adiscon.com> wrote:
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> I have a configured central log collector using rsyslog.
>>> A few
>>> >>>>>> of
>>> >>>>>>>>>> the
>>> >>>>>>>>>>>>>> devices forwarding their logs are appliances that have no
>>> >>>>>>>>>>> configuration
>>> >>>>>>>>>>>>>> options other than the IP forwarding address and
>>> protocol. I
>>> >>>>>> cannot
>>> >>>>>>>>>>>>> control
>>> >>>>>>>>>>>>>> what file names are being sent.
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> Unfortunately, they are sending unintelligible file names
>>> with
>>> >>>>>>>>>>> characters
>>> >>>>>>>>>>>>>> that normally would be escaped. Is there any way I can
>>> control
>>> >>>>>> or
>>> >>>>>>>>>>> alter
>>> >>>>>>>>>>>>> the
>>> >>>>>>>>>>>>>> incoming file name to normalize it to avoid these odd
>>> >>>>>> characters?
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> For example, could I establish a character map that maps
>>> the
>>> >>>>>>>>>> unallowed
>>> >>>>>>>>>>>>>> character to something acceptable?
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> thanks,
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> *Scott Slattery*
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> *Sr. Systems & Cloud Architect*
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> motorolasolutions.com
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> *O: 602.529.822*
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> --
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> *For more information on how and why we collect your
>>> personal
>>> >>>>>>>>>>>>>> information, please visit our Privacy Policy
>>> >>>>>>>>>>>>>> <
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>> >>>>>>>>>>>>>> .*
>>> >>>>>>>>>>>>>> _______________________________________________
>>> >>>>>>>>>>>>>> rsyslog mailing list
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
>>> >>>>>>>>>>>>>> What's up with rsyslog? Follow
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
>>> >>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
>>> ARCHIVED
>>> >> by
>>> >>>>>> a
>>> >>>>>>>>>>> myriad
>>> >>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>> POST
>>> >>>>>> if
>>> >>>>>>>> you
>>> >>>>>>>>>>>>> DON'T LIKE THAT.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> --
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> *For more information on how and why we collect your
>>> personal
>>> >>>>>>>>>>>> information, please visit our Privacy Policy
>>> >>>>>>>>>>>> <
>>> >>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>> >>>>>>>>>>>> .*
>>> >>>>>>>>>>>> _______________________________________________
>>> >>>>>>>>>>>> rsyslog mailing list
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
>>> >>>>>>>>>>>> What's up with rsyslog? Follow
>>> >>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
>>> >>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
>>> ARCHIVED by
>>> >> a
>>> >>>>>>>>>> myriad
>>> >>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>> POST
>>> >> if
>>> >>>>>> you
>>> >>>>>>>>>>> DON'T LIKE THAT.
>>> >>>>>>>>>>
>>> >>>>>>>>>> --
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> *For more information on how and why we collect your personal
>>> >>>>>>>>>> information, please visit our Privacy Policy
>>> >>>>>>>>>> <
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>> >>>>>>>>>>> .*
>>> >>>>>>>>>> _______________________________________________
>>> >>>>>>>>>> rsyslog mailing list
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
>>> >>>>>>>>>> What's up with rsyslog? Follow
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
>>> >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>> by a
>>> >>>>>> myriad
>>> >>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>> POST if
>>> >>>>>> you
>>> >>>>>>>>>> DON'T LIKE THAT.
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>
>>> >>>>>>> --
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> *For more information on how and why we collect your personal
>>> >>>>>>> information, please visit our Privacy Policy
>>> >>>>>>> <
>>> >>>>>>
>>> >>>>
>>> >>
>>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>> >>>>>>> .*
>>> >>>>>>> _______________________________________________
>>> >>>>>>> rsyslog mailing list
>>> >>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=7hc9b2gSfvsfQzXvsRVXqGeP4Rz_tJtrykpjqkGBIdk&e=
>>> >>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=bDybhvts9KqI-WEfMHo7O40ulQK5M6Tg4tnKNRRIhE8&e=
>>> >>>>>>> What's up with rsyslog? Follow
>>> >>>>>>
>>> >>>>
>>> >>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=zPhpcUPlK_CWLiU75_WwjyQGz5mFTiXc1yyBIfppkrI&e=
>>> >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> >>>> myriad
>>> >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>> if you
>>> >>>>>> DON'T LIKE THAT.
>>> >>>>>>
>>> >>>>>
>>> >>>>>
>>> >>>
>>> >>>
>>> >
>>> >
>>
>>
> *For more information on how and why we collect your personal information,
> please visit our Privacy Policy
> <https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Altering forwarded logfile names [ In reply to ]

rsyslog at lists

Mar 26, 2021, 1:07 AM

Post #29 of 30 (545 views)

Permalink

ahh, I missed that. A bit longer explination:

before the date, the message should start out with '<number>' or '<number>1'
depending on which RFC it's following (from the format of the date, this should
have the 1 as required by RFC5424) without that it's triggering the non-standard
extension that rsyslog supports that allows you to send embedded newlines in the
message.

with it starting with 2021 (which would be detected to be the message length),
that would explain why the newlines are being ignored.

disabling this feature as rainer suggests may fix it.

still a problem with the sender violating the RFC, just combined with a
extension of the syslog protocol that rsyslog support to make a set of odd
symptoms.

David Lang

On Fri, 26 Mar 2021, Rainer Gerhards wrote:

> Date: Fri, 26 Mar 2021 08:55:02 +0100
> From: Rainer Gerhards <rgerhards@hq.adiscon.com>
> To: Scott Slattery <scott.slattery@motorolasolutions.com>
> Cc: David Lang <david@lang.hm>, rsyslog-users <rsyslog@lists.adiscon.com>,
> mariusz.kruk@safecomp.com
> Subject: Re: [rsyslog] Altering forwarded logfile names
>
> I think I guess what is going on. Probably a message starts, without pri,
> with a number. If so, that triggers octet-counted framing, because that's
> what it really indicates. So the number becomes the message length.
>
> Try param
>
> supportOctedtCountedFraming="off"
>
> Doc:
>
> https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html
>
> Rainer
>
> El jue, 25 mar 2021 a las 23:37, Scott Slattery (<
> scott.slattery@motorolasolutions.com>) escribió:
>
>> Upon further review, I'm seeing packet lengths from 4434 down to 111. It
>> appears once the packet length exceeds 2919 the trailing 0x0a is missing
>> from any remaining packets larger than 2919.
>>
>> If I sort by packet size, it jumps from 2919 to 2974 then all further line
>> feed character is missing.
>>
>> *Scott Slattery*
>>
>> *Sr. Systems & Cloud Architect*
>>
>> *Cloud, Compute, Information & Architecture Team*
>>
>> motorolasolutions.com
>>
>> *O: 602.529.8226*
>>
>> *E*: Scott.Slattery@MotorolaSolutions.com
>>
>>
>>
>>
>> On Thu, Mar 25, 2021 at 3:24 PM Scott Slattery <
>> scott.slattery@motorolasolutions.com> wrote:
>>
>>>
>>> *Rsyslogd version:*
>>> rsyslogd 8.24.0-52.amzn2, compiled with:
>>> PLATFORM: x86_64-koji-linux-gnu
>>> PLATFORM (lsb_release -d):
>>> FEATURE_REGEXP: Yes
>>> GSSAPI Kerberos 5 support: Yes
>>> FEATURE_DEBUG (debug build, slow code): No
>>> 32bit Atomic operations supported: Yes
>>> 64bit Atomic operations supported: Yes
>>> memory allocator: system default
>>> Runtime Instrumentation (slow code): No
>>> uuid support: Yes
>>> Number of Bits in RainerScript integers: 64
>>>
>>> Here is my config:
>>> $MaxMessageSize 64k
>>> $EscapeControlCharactersOnReceive on
>>>
>>> module(load="imuxsock")
>>> module(load="imjournal" StateFile="imjournal.state")
>>> module(load="imklog")
>>> module(load="immark")
>>> module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat"
>>> DirCreateMode="0705" FileCreateMode="0704")
>>> module(load="impstats" interval="600" log.syslog="off" severity="7"
>>> log.file="/var/log/impstats.log")
>>>
>>> $DynaFileCacheSize 100
>>> global(workDirectory="/var/lib/rsyslog" localHostname="cloudlogmaster01")
>>>
>>> $MainMsgQueueSize 200000
>>> $MainMsgQueueHighWaterMark 120000
>>> $MainMsgQueueDiscardMark 160000
>>> $MainMsgQueueWorkerThreads 100
>>> $RulesetCreateMainQueue on
>>>
>>> $imjournalRatelimitInterval 0
>>> $imjournalRatelimitBurst 0
>>> #### RULES ####
>>>
>>> template(name="debug" type="string" string="Debug line with all
>>> properties:\nFROMHOST: '%FROMHOST%'\n, FROMHOST-IP: '%fromhost-ip%'\n,
>>> HOSTNAME: '%HOSTNAME%'\n, PRI: %PRI%,\nSYSLOGTAG '%syslogtag%'\n,
>>> PROGRAMNAME: '%programname%'\n, APP-NAME: '%app-name%'\n, PROCID:
>>> '%PROCID%'\n, MSGID: '%MSGID%'\n, TIMESTAMP: '%TIMESTAMP%'\n,
>>> STRUCTURED-DATA: '%STRUCTURED-DATA%'\n, msg: '%msg%'\nRAWMSG:
>>> '%rawmsg%',\n\nRAWMSG-AFTER-PRI: '%rawmsg-after-pri%'\nJSONMESG:
>>> '%jsonmesg%'\n")
>>>
>>> template(name="DynRemoteLogFile" type="string"
>>> string="/splunklog/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
>>>
>>> template(name="MalformedMsgFormatter" type="string"
>>> string="%timegenerated% %fromhost% %rawmsg:::drop-last-lf%\n")
>>> template(name="DynRemoteMalformed" type="string"
>>> string="/splunklog/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-oag-audit.log")
>>>
>>> template(name="DynRemoteLogFile" type="list") {
>>> constant(value="/splunklog/remote/")
>>> property(name="FROMHOST")
>>> constant(value="-")
>>> property(name="FROMHOST-IP")
>>> constant(value="/")
>>> property(name="$YEAR")
>>> constant(value="-")
>>> property(name="$MONTH")
>>> constant(value="-")
>>> property(name="$DAY")
>>> constant(value="-")
>>> property(name="syslogtag" position.from="1" position.to="32")
>>> constant(value="-")
>>> property(name="app-name" controlcharacters="drop")
>>> constant(value=".log")
>>> }
>>>
>>> ruleset(name="RemoteServer"){
>>> if ($fromhost-ip startswith '10.41.102') then {
>>> action(type="omfile" dynafile="DynRemoteMalformed"
>>> template="MalformedMsgFormatter")
>>> stop
>>> }
>>>
>>> action(type="omfile" dynaFile="DynRemoteLogFile")
>>> }
>>>
>>>
>>> # module imtcp
>>> module(load="imtcp" KeepAlive="on")
>>> input(type="imtcp" port="10514" ruleset="RemoteServer")
>>>
>>>
>>> #kern.* /dev/console
>>> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>>> authpriv.* /var/log/secure
>>> mail.* /var/log/maillog
>>> cron.* /var/log/cron
>>> *.emerg :omusrmsg:*
>>> uucp,news.crit /var/log/spooler
>>> local7.* /var/log/boot.log
>>>
>>> *Scott Slattery*
>>>
>>> *Sr. Systems & Cloud Architect*
>>>
>>> *Cloud, Compute, Information & Architecture Team*
>>>
>>> motorolasolutions.com
>>>
>>> *O: 602.529.8226*
>>>
>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>
>>>
>>>
>>>
>>> On Thu, Mar 25, 2021 at 3:20 PM David Lang <david@lang.hm> wrote:
>>>
>>>> what version of rsyslog are you running. can you post your full config?
>>>>
>>>> if you are receiving via TCP and it's not splitting the logs based on
>>>> newlines,
>>>> something very odd is happening.
>>>>
>>>> David Lang
>>>>
>>>> On Thu, 25 Mar 2021, Scott Slattery wrote:
>>>>
>>>>> Date: Thu, 25 Mar 2021 15:07:22 -0700
>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>>>> To: David Lang <david@lang.hm>
>>>>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
>>>>> rsyslog-users <rsyslog@lists.adiscon.com>,
>>>> mariusz.kruk@safecomp.com
>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>>>
>>>>> So I'm looking at the pcap and on the first log record the header
>>>>> information looks fine and the record terminated by 0x0A line feed.
>>>>>
>>>>> When I look at the record containing this record written to the rsyslog
>>>>> file, the record length is 65586 (64k) long which is what I think you
>>>> were
>>>>> referring to earlier. Why would rsyslog write such a long record? I
>>>>> initially thought my data may be getting truncated so I updated
>>>>> '$MaxMessageSize 64k'. Any suggestion how I fix this behavior.
>>>>>
>>>>> I can confirm that each log record in the pcap is terminated by 0x0A
>>>> and
>>>>> looks in tact so must be something I need to do on rsyslog to fix this.
>>>>>
>>>>> *Scott Slattery*
>>>>>
>>>>> *Sr. Systems & Cloud Architect*
>>>>>
>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>
>>>>> motorolasolutions.com
>>>>>
>>>>> *O: 602.529.8226*
>>>>>
>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Mar 25, 2021 at 1:15 PM David Lang <david@lang.hm> wrote:
>>>>>
>>>>>> you may want to capture with -X so that it decodes it into hex and
>>>> you can
>>>>>> see
>>>>>> newlines vs linefeeds
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>> On Thu, 25 Mar 2021, Scott Slattery wrote:
>>>>>>
>>>>>>> Date: Thu, 25 Mar 2021 12:47:43 -0700
>>>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>>>>>> To: David Lang <david@lang.hm>
>>>>>>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
>>>>>>> rsyslog-users <rsyslog@lists.adiscon.com>,
>>>> mariusz.kruk@safecomp.com
>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>>>>>
>>>>>>> Thanks David, I just grabbed my first capture and am looking at it.
>>>> I'm
>>>>>>> using TCP vs UDP but your comment makes sense. I'll certainly share
>>>> with
>>>>>>> you my observations once I have some data to refer to. This has been
>>>> a
>>>>>>> learning experience. It's also the first time I've ever seen such an
>>>>>> issue
>>>>>>> with rsyslog so it's a bit of a puzzle.
>>>>>>>
>>>>>>> *Scott Slattery*
>>>>>>>
>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>
>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>
>>>>>>> motorolasolutions.com
>>>>>>>
>>>>>>> *O: 602.529.8226*
>>>>>>>
>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Mar 25, 2021 at 12:32 PM David Lang <david@lang.hm> wrote:
>>>>>>>
>>>>>>>> the rawmsg field in the debugformat output shows exactly what
>>>> rsyslog is
>>>>>>>> seeing.
>>>>>>>>
>>>>>>>> the reason I asked you to check multiple entries is that if rsyslog
>>>> does
>>>>>>>> not see
>>>>>>>> the separator (due to either multiple messages in one UDP packet, or
>>>>>>>> missing
>>>>>>>> newlines in a TCP stream) it will combine what are intended as
>>>> multiple
>>>>>>>> messages
>>>>>>>> together into one message as far as rsyslog is concerned, and only
>>>> break
>>>>>>>> it into
>>>>>>>> multiple messages when it hits the configured maxmessagesize size.
>>>> That
>>>>>>>> breaks
>>>>>>>> things at arbitrary points in the message, so as you look at
>>>> multiple
>>>>>>>> messages,
>>>>>>>> they will not all start the same way.
>>>>>>>>
>>>>>>>> a tcpdump -s0 -A -n (with the appropriate port/ip filter) will also
>>>> show
>>>>>>>> exactly
>>>>>>>> what's happening over the wire.
>>>>>>>>
>>>>>>>> David Lang
>>>>>>>>
>>>>>>>> On Thu, 25 Mar 2021, Scott Slattery wrote:
>>>>>>>>
>>>>>>>>> Date: Thu, 25 Mar 2021 12:23:03 -0700
>>>>>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>>>>>>>> To: Rainer Gerhards <rgerhards@hq.adiscon.com>
>>>>>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
>>>>>> david@lang.hm
>>>>>>>>> ,
>>>>>>>>> mariusz.kruk@safecomp.com
>>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>>>>>>>
>>>>>>>>> Hey Guys, before I consume more of your time, I'm capturing the
>>>> packets
>>>>>>>>> from the source device to determine if any of the payload is lost
>>>> on
>>>>>> its
>>>>>>>>> way to the rsyslog collector. After I analyze further, I'll share
>>>> my
>>>>>>>>> observations. I'm hoping nothing is getting dropped over the wire.
>>>>>> Thanks
>>>>>>>>> for your patience and help so far. I expect to have additional
>>>> details
>>>>>>>>> shortly.
>>>>>>>>>
>>>>>>>>> *Scott Slattery*
>>>>>>>>>
>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>>
>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>>
>>>>>>>>> motorolasolutions.com
>>>>>>>>>
>>>>>>>>> *O: 602.529.8226*
>>>>>>>>>
>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Mar 25, 2021 at 12:51 AM Rainer Gerhards <
>>>>>>>> rgerhards@hq.adiscon.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Sorry if I didn't catch it, but did you post the configuration?
>>>> If
>>>>>>>>>> not, it would probably be useful.
>>>>>>>>>>
>>>>>>>>>> Looking at the message properties, this looks like the message was
>>>>>>>>>> received via UDP but had a LF inside it which was intended as a
>>>> frame
>>>>>>>>>> separator. If so, that is another bug in the vendor's
>>>> implementation.
>>>>>>>>>> Framing is used for TCP, but not UDP. For UDP it is strictly one
>>>>>>>>>> message per package.
>>>>>>>>>>
>>>>>>>>>> HTH,
>>>>>>>>>> Rainer
>>>>>>>>>>
>>>>>>>>>> El mié, 24 mar 2021 a las 22:23, Scott Slattery via rsyslog
>>>>>>>>>> (<rsyslog@lists.adiscon.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>> From what I can discern since there are lots of transactions,
>>>> there
>>>>>>>> seem
>>>>>>>>>> to
>>>>>>>>>>> be multiple payload formats. For all that I have so far
>>>> analyzed, the
>>>>>>>>>> date
>>>>>>>>>>> truncation issue is the same regardless of the payload received.
>>>> Like
>>>>>>>>>> you,
>>>>>>>>>>> I've never used mmnormalize but I will look at it as an option.
>>>> I've
>>>>>>>>>>> reached out to the product support team to better understand if
>>>> there
>>>>>>>> is
>>>>>>>>>> a
>>>>>>>>>>> appliance-side misconfiguration and something that can be
>>>> corrected
>>>>>>>> there
>>>>>>>>>>> or if I need to make adjustments on the rsyslog side to clean up
>>>> the
>>>>>>>>>>> records.
>>>>>>>>>>>
>>>>>>>>>>> What I meant is that there are multiple message formats (payload
>>>>>> data)
>>>>>>>>>> but
>>>>>>>>>>> there seems to be a consistent truncation of the date field just
>>>> as
>>>>>> you
>>>>>>>>>> saw
>>>>>>>>>>> in the app-name tag field.
>>>>>>>>>>>
>>>>>>>>>>> I appreciate your ongoing feedback and help. This issue is new
>>>> to me
>>>>>>>> and
>>>>>>>>>>> will need a creative solution if the vendor has no answers.
>>>>>>>>>>>
>>>>>>>>>>> Many thanks,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *Scott Slattery*
>>>>>>>>>>>
>>>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>>>>
>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>>>>
>>>>>>>>>>> motorolasolutions.com
>>>>>>>>>>>
>>>>>>>>>>> *O: 602.529.8226*
>>>>>>>>>>>
>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Mar 24, 2021 at 2:14 PM David Lang <david@lang.hm>
>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> when you say 'each is incorrect but the same format'
>>>>>>>>>>>>
>>>>>>>>>>>> does that mean that every log has the year missing? or that
>>>> every
>>>>>> log
>>>>>>>>>> is
>>>>>>>>>>>> combining the logs together?
>>>>>>>>>>>>
>>>>>>>>>>>> I'll note that it's possible to define a custom parser using
>>>> the
>>>>>>>>>>>> mmnormalize
>>>>>>>>>>>> library and add it to the parsing stack. I helped define the
>>>> need
>>>>>> for
>>>>>>>>>> such
>>>>>>>>>>>> a
>>>>>>>>>>>> capability, but haven't used it yet.
>>>>>>>>>>>>
>>>>>>>>>>>> David Lang
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, 24 Mar 2021, Scott Slattery wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Date: Wed, 24 Mar 2021 08:58:07 -0700
>>>>>>>>>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>>>>>>>>>>>> To: mariusz.kruk@safecomp.com
>>>>>>>>>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
>>>>>>>>>> david@lang.hm
>>>>>>>>>>>>>
>>>>>>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, all the logs are coming in this way. There are three
>>>> separate
>>>>>>>>>> logs
>>>>>>>>>>>>> being sent and each is incorrect in its own right but the same
>>>>>> format
>>>>>>>>>>>>> behavior is consistent across each of the three logs. I will
>>>> look
>>>>>>>>>> further
>>>>>>>>>>>>> into the suggestions you have provided and have already
>>>> reached out
>>>>>>>>>> to
>>>>>>>>>>>> the
>>>>>>>>>>>>> appliance support team and will raise the nonconformance with
>>>> the
>>>>>>>>>> RFC.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This does provide me some valuable guidance and I thank each
>>>> of you
>>>>>>>>>> for
>>>>>>>>>>>>> your feedback.
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Scott Slattery*
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>>>>>>
>>>>>>>>>>>>> motorolasolutions.com
>>>>>>>>>>>>>
>>>>>>>>>>>>> *O: 602.529.8226*
>>>>>>>>>>>>>
>>>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com>
>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> There's something strange with this log entry. Notice that it
>>>>>> looks
>>>>>>>>>> as
>>>>>>>>>>>> if
>>>>>>>>>>>>>> it were more than one log entries joined with LF character
>>>>>> (escaped
>>>>>>>>>> on
>>>>>>>>>>>>>> input as #012). And all "further" entries start with a - still
>>>>>>>>>>>>>> nonconformant to the RFC but "fuller" - timestamp containing
>>>>>>>>>> year.It's
>>>>>>>>>>>> the
>>>>>>>>>>>>>> leading event that seems to be trimmed in the front somehow.
>>>> Do
>>>>>> all
>>>>>>>>>>>> events
>>>>>>>>>>>>>> arrive like this?Wys?ano z telefonu Samsung<div>
>>>>>>>>>>>>>> </div><div>
>>>>>>>>>>>>>> </div><div>-------- Original message
>>>>>>>>>>>>>> --------</div><div>From: Scott Slattery via rsyslog <
>>>>>>>>>>>>>> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021 01:52
>>>>>>>>>>>>>> (GMT+01:00) </div><div>To: David Lang <david@lang.hm>
>>>>>>>>>> </div><div>Cc:
>>>>>>>>>>>>>> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott
>>>>>>>>>> Slattery
>>>>>>>>>>>> via
>>>>>>>>>>>>>> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re:
>>>>>>>>>> [rsyslog]
>>>>>>>>>>>>>> Altering forwarded logfile names </div><div>
>>>>>>>>>>>>>> </div>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi David, fortunately I had already done this. I'm including
>>>> an
>>>>>>>>>> actual
>>>>>>>>>>>> log
>>>>>>>>>>>>>> entry but have anonymized the data to keep the actual user and
>>>>>> email
>>>>>>>>>>>>>> address confidential:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Debug line with all properties:
>>>>>>>>>>>>>> FROMHOST: 'ause1oagatst02.aws.mycompany.com'
>>>>>>>>>>>>>> , fromhost-ip: '10.41.102.143'
>>>>>>>>>>>>>> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
>>>>>>>>>>>>>> , PRI: 13,
>>>>>>>>>>>>>> syslogtag '03-23T16:'
>>>>>>>>>>>>>> , programname: '03-23T16'
>>>>>>>>>>>>>> , APP-NAME: '03-23T16'
>>>>>>>>>>>>>> , PROCID: '-'
>>>>>>>>>>>>>> , MSGID: '-'
>>>>>>>>>>>>>> , TIMNESTAMP: 'Mar 23 21:47:13'
>>>>>>>>>>>>>> , STRUCTURED-DATA: '-'
>>>>>>>>>>>>>> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
>>>>>>>>>>>> ACCESS_GATEWAY
>>>>>>>>>>>>>> ACCESS AUTHZ SESSION INFO USER_SESSION
>>>>>>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>> SUBJECT="
>>>>>>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>>>>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>>>>>>>> apigniodashboard-uat.mycompany.com"
>>>>>>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>>>>>>>>>>> REMOTE_IP="10.44.65.38"
>>>>>>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>>>>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>>>>>>>> Safari/537.36"]
>>>>>>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>>>>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>>>>> POLICY
>>>>>>>>>>>> INFO
>>>>>>>>>>>>>> USER_AUTHZ
>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
>>>>>> RESOURCE="/_dash-dependencies"
>>>>>>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED"
>>>> DURATION="0"
>>>>>>>>>>>> APP="Ignio
>>>>>>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW"
>>>> REASON="N/A -
>>>>>>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
>>>>>> oag_username=
>>>>>>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>>>>>>>>>>>> apigniodashboard-uat.mycompany.com
>>>>>>>>>>>>>> UserName=Joe.User@mycompantions.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>>>>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
>>>> Mac
>>>>>>>>>> OS X
>>>>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>>>>> Chrome/88.0.4324.192
>>>>>>>>>>>>>> Safari/537.36 creationTime=1616536016811
>>>>>> maxInactiveInterval=3600000
>>>>>>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>>>>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
>>>> Intel
>>>>>>>>>> Mac
>>>>>>>>>>>> OS X
>>>>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>>>>> Chrome/88.0.4324.192
>>>>>>>>>>>>>> Safari/537.36"] allow access to
>>>>>>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>>>>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>>>>>>>>> SESSION
>>>>>>>>>>>> INFO
>>>>>>>>>>>>>> USER_SESSION
>>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>>>>>>>> SUBJECT="Joe.User@mycompan'
>>>>>>>>>>>>>> *escaped msg*: '47:20.708-05:00
>>>> AUSE1OAGATST02.aws.mycompany.com
>>>>>>>>>>>>>> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
>>>>>>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>> SUBJECT="
>>>>>>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>>>>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>>>>>>>> apigniodashboard-uat.mycompany.com"
>>>>>>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>>>>>>>>>>> REMOTE_IP="10.44.65.38"
>>>>>>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>>>>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>>>>>>>> Safari/537.36"]
>>>>>>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>>>>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>>>>> POLICY
>>>>>>>>>>>> INFO
>>>>>>>>>>>>>> USER_AUTHZ
>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
>>>>>> RESOURCE="/_dash-dependencies"
>>>>>>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED"
>>>> DURATION="0"
>>>>>>>>>>>> APP="Ignio
>>>>>>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW"
>>>> REASON="N/A -
>>>>>>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
>>>>>> oag_username=
>>>>>>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>>>>>>>>>>>> apigniodashboard-uat.mycompany.com
>>>>>>>>>>>>>> UserName=Joe.User@mycompantions.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>>>>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
>>>> Mac
>>>>>>>>>> OS X
>>>>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>>>>> Chrome/88.0.4324.192
>>>>>>>>>>>>>> Safari/537.36 creationTime=1616536016811
>>>>>> maxInactiveInterval=3600000
>>>>>>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>>>>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
>>>> Intel
>>>>>>>>>> Mac
>>>>>>>>>>>> OS X
>>>>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>>>>> Chrome/88.0.4324.192
>>>>>>>>>>>>>> Safari/537.36"] allow access to
>>>>>>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>>>>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>>>>>>>>> SESSION
>>>>>>>>>>>> INFO
>>>>>>>>>>>>>> USER_SESSION
>>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>>>>>>>> SUBJECT="Joe.User@mycompan'
>>>>>>>>>>>>>> *rawmsg*: '03-23T16:47:20.708-05:00
>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com
>>>>>>>>>>>>>> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
>>>>>>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>> SUBJECT="
>>>>>>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>>>>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>>>>>>>> apigniodashboard-uat.mycompany.com"
>>>>>>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>>>>>>>>>>> REMOTE_IP="10.44.65.38"
>>>>>>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>>>>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>>>>>>>>>> Safari/537.36"]
>>>>>>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>>>>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>>>>> POLICY
>>>>>>>>>>>> INFO
>>>>>>>>>>>>>> USER_AUTHZ
>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
>>>>>> RESOURCE="/_dash-dependencies"
>>>>>>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED"
>>>> DURATION="0"
>>>>>>>>>>>> APP="Ignio
>>>>>>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>>>>>>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW"
>>>> REASON="N/A -
>>>>>>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
>>>>>> oag_username=
>>>>>>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>>>>>>>>>>>> apigniodashboard-uat.mycompany.com
>>>>>>>>>>>>>> UserName=Joe.User@mycompantions.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>>>>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel
>>>> Mac
>>>>>>>>>> OS X
>>>>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>>>>> Chrome/88.0.4324.192
>>>>>>>>>>>>>> Safari/537.36 creationTime=1616536016811
>>>>>> maxInactiveInterval=3600000
>>>>>>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>>>>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
>>>> Intel
>>>>>>>>>> Mac
>>>>>>>>>>>> OS X
>>>>>>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>>>>> Chrome/88.0.4324.192
>>>>>>>>>>>>>> Safari/537.36"] allow access to
>>>>>>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>>>>>>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ
>>>>>>>>>> SESSION
>>>>>>>>>>>> INFO
>>>>>>>>>>>>>> USER_SESSION
>>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>>>>>>>>>>>> SUBJECT="Joe.User@mycompan'
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> By inspecting the rawmsg, I can see that field four
>>>>>>>>>> (space-delimited)
>>>>>>>>>>>>>> indicates this is the ACCESS log. So if I were able to
>>>> extract the
>>>>>>>>>> log
>>>>>>>>>>>>>> identifier from the msg, I could then write all access logs
>>>> to the
>>>>>>>>>> same
>>>>>>>>>>>>>> daily file. There are other formats as well from the same
>>>> device
>>>>>>>>>> but the
>>>>>>>>>>>>>> idea is the same.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Scott Slattery*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> motorolasolutions.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *O: 602.529.8226*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm>
>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> the source logfile name is not included in the payload by the
>>>>>>>>>> syslog
>>>>>>>>>>>>>> spec.
>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>> may be in the case of your appliance, but we would need to
>>>> see a
>>>>>>>>>> sample
>>>>>>>>>>>>>>> log to
>>>>>>>>>>>>>>> understand ho to parse it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> based on your template, you are using app-name, which may be
>>>>>> listed
>>>>>>>>>>>>>>> separtely if
>>>>>>>>>>>>>>> it's a RFC5424 format log, or may be part of the syslog tag
>>>> if
>>>>>>>>>> it's a
>>>>>>>>>>>>>>> RFC3164
>>>>>>>>>>>>>>> format log over the wire (neither format has a way to
>>>> specify a
>>>>>>>>>> source
>>>>>>>>>>>>>> log
>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>> by default)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> you can look at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
>>>>>>>>>>>>>>> and see the *-cc
>>>>>>>>>>>>>>> options that you could apply to the app-name to eliminate
>>>> control
>>>>>>>>>>>>>>> characters.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Again, we really need to see the original log message to
>>>>>> understand
>>>>>>>>>>>>>> what's
>>>>>>>>>>>>>>> what.
>>>>>>>>>>>>>>> Please log it with the templateRSYSLOG_DebugFormat so we can
>>>> see
>>>>>>>>>>>> exactly
>>>>>>>>>>>>>>> what is
>>>>>>>>>>>>>>> sent over the wire and how rsyslog has parsed it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> David Lang
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
>>>>>>>>>>>>>>>> From: Scott Slattery via rsyslog <rsyslog@lists.adiscon.com
>>>>>
>>>>>>>>>>>>>>>> To: John Chivian <jchivian@chivian.com>
>>>>>>>>>>>>>>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
>>>>>>>>>>>>>>>> rsyslog-users <rsyslog@lists.adiscon.com>
>>>>>>>>>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks, John, let me try to clarify what I mean.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Normally when I forward from a remote server to the central
>>>> log
>>>>>>>>>>>>>> server, I
>>>>>>>>>>>>>>>> can include a tag that can then be used to determine the
>>>> file
>>>>>>>>>> name I
>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>> on the central server. Since I have no real way to include
>>>> this
>>>>>>>>>> tag
>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>> the appliance, this is not an option.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm looking for a way of inspecting the incoming packets to
>>>>>>>>>>>> determining
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> source logfile name (which is included in the payload) and
>>>> use
>>>>>>>>>> that
>>>>>>>>>>>>>>>> filename on the target central server. Since there are
>>>> multiple
>>>>>>>>>> logs
>>>>>>>>>>>>>>> being
>>>>>>>>>>>>>>>> sent (access, audit, monitor, etc.), I'd like to segregate
>>>> these
>>>>>>>>>> into
>>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>>> own files. I'm already using a template with the host
>>>>>> information
>>>>>>>>>> to
>>>>>>>>>>>>>>>> dynamically create the file names. I just don't know how I
>>>> can
>>>>>> go
>>>>>>>>>>>>>> beyond
>>>>>>>>>>>>>>>> this to also include the source logname.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Here's the template I'm using. It works for all other hosts
>>>>>> where
>>>>>>>>>> I
>>>>>>>>>>>> can
>>>>>>>>>>>>>>>> configure the tag but I get garbage names from the
>>>> appliance. I
>>>>>>>>>> had
>>>>>>>>>>>>>> hoped
>>>>>>>>>>>>>>>> that the appliance included some standard syslog tags but it
>>>>>>>>>> doesn't
>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>> so.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> template(name="DynRemoteLogFile" type="string"
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *Scott Slattery*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> motorolasolutions.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *O: 602.529.8226*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <
>>>>>>>>>> jchivian@chivian.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Your use of the term “file name” is confusing. When
>>>> senders
>>>>>>>>>> deliver
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> rsyslog over the network there is no exchange of files or
>>>>>>>>>> filenames,
>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>> packets of information. Those packets are expected to be
>>>> in a
>>>>>>>>>> format
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> syslog understands such that useful information (header
>>>>>> elements
>>>>>>>>>> and
>>>>>>>>>>>>>>>>> message body) may be parsed from them. If you as the
>>>> rsyslog
>>>>>>>>>> admin
>>>>>>>>>>>>>>> choose
>>>>>>>>>>>>>>>>> to use some of that header information to compose
>>>> filenames for
>>>>>>>>>>>> output
>>>>>>>>>>>>>>>>> files, then yes you are sort of at the mercy of the senders
>>>>>>>>>> content
>>>>>>>>>>>>>>>>> (especially if the sender doesn’t follow the syslog rules).
>>>>>>>>>> However,
>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>> are functions in the advanced syntax that can be used to
>>>>>> perform
>>>>>>>>>> the
>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>>>> of character replacements you’re talking about.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It is common practice to use the syslog header/rsyslog
>>>> property
>>>>>>>>>>>>>> element
>>>>>>>>>>>>>>>>> called “hostname” for just such purposes. Is this what
>>>> you’re
>>>>>>>>>>>> talking
>>>>>>>>>>>>>>>>> about? You’d have to provide your configuration for real
>>>>>>>>>> analysis,
>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> least the part you perceive to be responsible for the
>>>> problem.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
>>>>>>>>>>>>>>>>> rsyslog@lists.adiscon.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I have a configured central log collector using rsyslog.
>>>> A few
>>>>>>>>>> of
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> devices forwarding their logs are appliances that have no
>>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>>>>> options other than the IP forwarding address and
>>>> protocol. I
>>>>>>>>>> cannot
>>>>>>>>>>>>>>>>> control
>>>>>>>>>>>>>>>>>> what file names are being sent.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Unfortunately, they are sending unintelligible file names
>>>> with
>>>>>>>>>>>>>>> characters
>>>>>>>>>>>>>>>>>> that normally would be escaped. Is there any way I can
>>>> control
>>>>>>>>>> or
>>>>>>>>>>>>>>> alter
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> incoming file name to normalize it to avoid these odd
>>>>>>>>>> characters?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> For example, could I establish a character map that maps
>>>> the
>>>>>>>>>>>>>> unallowed
>>>>>>>>>>>>>>>>>> character to something acceptable?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Scott Slattery*
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> motorolasolutions.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *O: 602.529.822*
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *For more information on how and why we collect your
>>>> personal
>>>>>>>>>>>>>>>>>> information, please visit our Privacy Policy
>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>>>>>>>>>>>>>> .*
>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
>>>>>>>>>>>>>>>>>> What's up with rsyslog? Follow
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
>>>>>>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
>>>> ARCHIVED
>>>>>> by
>>>>>>>>>> a
>>>>>>>>>>>>>>> myriad
>>>>>>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>> POST
>>>>>>>>>> if
>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>> DON'T LIKE THAT.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *For more information on how and why we collect your
>>>> personal
>>>>>>>>>>>>>>>> information, please visit our Privacy Policy
>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>>>>>>>>>>>> .*
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
>>>>>>>>>>>>>>>> What's up with rsyslog? Follow
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
>>>>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
>>>> ARCHIVED by
>>>>>> a
>>>>>>>>>>>>>> myriad
>>>>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>> POST
>>>>>> if
>>>>>>>>>> you
>>>>>>>>>>>>>>> DON'T LIKE THAT.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *For more information on how and why we collect your personal
>>>>>>>>>>>>>> information, please visit our Privacy Policy
>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>>>>>>>>>>> .*
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
>>>>>>>>>>>>>> What's up with rsyslog? Follow
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
>>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>> by a
>>>>>>>>>> myriad
>>>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>> POST if
>>>>>>>>>> you
>>>>>>>>>>>>>> DON'T LIKE THAT.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *For more information on how and why we collect your personal
>>>>>>>>>>> information, please visit our Privacy Policy
>>>>>>>>>>> <
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>>>>>>>>> .*
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=7hc9b2gSfvsfQzXvsRVXqGeP4Rz_tJtrykpjqkGBIdk&e=
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=bDybhvts9KqI-WEfMHo7O40ulQK5M6Tg4tnKNRRIhE8&e=
>>>>>>>>>>> What's up with rsyslog? Follow
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=zPhpcUPlK_CWLiU75_WwjyQGz5mFTiXc1yyBIfppkrI&e=
>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>>>> myriad
>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>>> if you
>>>>>>>>>> DON'T LIKE THAT.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>> *For more information on how and why we collect your personal information,
>> please visit our Privacy Policy
>> <https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: Altering forwarded logfile names [ In reply to ]

rsyslog at lists

Mar 26, 2021, 8:15 AM

Post #30 of 30 (545 views)

Permalink

Good morning David and Rainer - I want to thank you both for your valuable
insights and feedback. I added the recommended parameter
SupportOctetCountedFraming="off" and it seems to have fixed the frame
misalignment issue and my logs are so far looking as I would have expected.
Not sure how long it would have taken me to find this piece of information
without your assistance so thank you again! I'm a happy camper.

I suppose we are all in one camp or another but I've always been a fan of
rsyslog and have used it many years. The support folks were arguing that I
should dump rsyslog because it introduces "problems" but without
empirical data to backup their claim. Once again, rsyslog wins and I'm very
happy about this.

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: Scott.Slattery@MotorolaSolutions.com

On Fri, Mar 26, 2021 at 12:55 AM Rainer Gerhards <rgerhards@hq.adiscon.com>
wrote:

> I think I guess what is going on. Probably a message starts, without pri,
> with a number. If so, that triggers octet-counted framing, because that's
> what it really indicates. So the number becomes the message length.
>
> Try param
>
> supportOctedtCountedFraming="off"
>
> Doc:
>
> https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.rsyslog.com_doc_v8-2Dstable_configuration_modules_imtcp.html&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=cHrE5tBiCva4OTeUEFf7UbYs5AtP14aWiI4w4N6w9Bs&e=>
>
> Rainer
>
> El jue, 25 mar 2021 a las 23:37, Scott Slattery (<
> scott.slattery@motorolasolutions.com>) escribió:
>
>> Upon further review, I'm seeing packet lengths from 4434 down to 111. It
>> appears once the packet length exceeds 2919 the trailing 0x0a is missing
>> from any remaining packets larger than 2919.
>>
>> If I sort by packet size, it jumps from 2919 to 2974 then all further
>> line feed character is missing.
>>
>> *Scott Slattery*
>>
>> *Sr. Systems & Cloud Architect*
>>
>> *Cloud, Compute, Information & Architecture Team*
>>
>> motorolasolutions.com
>>
>> *O: 602.529.8226*
>>
>> *E*: Scott.Slattery@MotorolaSolutions.com
>>
>>
>>
>>
>> On Thu, Mar 25, 2021 at 3:24 PM Scott Slattery <
>> scott.slattery@motorolasolutions.com> wrote:
>>
>>>
>>> *Rsyslogd version:*
>>> rsyslogd 8.24.0-52.amzn2, compiled with:
>>> PLATFORM: x86_64-koji-linux-gnu
>>> PLATFORM (lsb_release -d):
>>> FEATURE_REGEXP: Yes
>>> GSSAPI Kerberos 5 support: Yes
>>> FEATURE_DEBUG (debug build, slow code): No
>>> 32bit Atomic operations supported: Yes
>>> 64bit Atomic operations supported: Yes
>>> memory allocator: system default
>>> Runtime Instrumentation (slow code): No
>>> uuid support: Yes
>>> Number of Bits in RainerScript integers: 64
>>>
>>> Here is my config:
>>> $MaxMessageSize 64k
>>> $EscapeControlCharactersOnReceive on
>>>
>>> module(load="imuxsock")
>>> module(load="imjournal" StateFile="imjournal.state")
>>> module(load="imklog")
>>> module(load="immark")
>>> module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat"
>>> DirCreateMode="0705" FileCreateMode="0704")
>>> module(load="impstats" interval="600" log.syslog="off" severity="7"
>>> log.file="/var/log/impstats.log")
>>>
>>> $DynaFileCacheSize 100
>>> global(workDirectory="/var/lib/rsyslog" localHostname="cloudlogmaster01")
>>>
>>> $MainMsgQueueSize 200000
>>> $MainMsgQueueHighWaterMark 120000
>>> $MainMsgQueueDiscardMark 160000
>>> $MainMsgQueueWorkerThreads 100
>>> $RulesetCreateMainQueue on
>>>
>>> $imjournalRatelimitInterval 0
>>> $imjournalRatelimitBurst 0
>>> #### RULES ####
>>>
>>> template(name="debug" type="string" string="Debug line with all
>>> properties:\nFROMHOST: '%FROMHOST%'\n, FROMHOST-IP: '%fromhost-ip%'\n,
>>> HOSTNAME: '%HOSTNAME%'\n, PRI: %PRI%,\nSYSLOGTAG '%syslogtag%'\n,
>>> PROGRAMNAME: '%programname%'\n, APP-NAME: '%app-name%'\n, PROCID:
>>> '%PROCID%'\n, MSGID: '%MSGID%'\n, TIMESTAMP: '%TIMESTAMP%'\n,
>>> STRUCTURED-DATA: '%STRUCTURED-DATA%'\n, msg: '%msg%'\nRAWMSG:
>>> '%rawmsg%',\n\nRAWMSG-AFTER-PRI: '%rawmsg-after-pri%'\nJSONMESG:
>>> '%jsonmesg%'\n")
>>>
>>> template(name="DynRemoteLogFile" type="string"
>>> string="/splunklog/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
>>>
>>> template(name="MalformedMsgFormatter" type="string"
>>> string="%timegenerated% %fromhost% %rawmsg:::drop-last-lf%\n")
>>> template(name="DynRemoteMalformed" type="string"
>>> string="/splunklog/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-oag-audit.log")
>>>
>>> template(name="DynRemoteLogFile" type="list") {
>>> constant(value="/splunklog/remote/")
>>> property(name="FROMHOST")
>>> constant(value="-")
>>> property(name="FROMHOST-IP")
>>> constant(value="/")
>>> property(name="$YEAR")
>>> constant(value="-")
>>> property(name="$MONTH")
>>> constant(value="-")
>>> property(name="$DAY")
>>> constant(value="-")
>>> property(name="syslogtag" position.from="1" position.to
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__position.to&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=EuEELua8r3xvmg1aJcuUo751CAwlGloSaQ1QbRPs1o8&e=>
>>> ="32")
>>> constant(value="-")
>>> property(name="app-name" controlcharacters="drop")
>>> constant(value=".log")
>>> }
>>>
>>> ruleset(name="RemoteServer"){
>>> if ($fromhost-ip startswith '10.41.102') then {
>>> action(type="omfile" dynafile="DynRemoteMalformed"
>>> template="MalformedMsgFormatter")
>>> stop
>>> }
>>>
>>> action(type="omfile" dynaFile="DynRemoteLogFile")
>>> }
>>>
>>>
>>> # module imtcp
>>> module(load="imtcp" KeepAlive="on")
>>> input(type="imtcp" port="10514" ruleset="RemoteServer")
>>>
>>>
>>> #kern.* /dev/console
>>> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>>> authpriv.* /var/log/secure
>>> mail.* /var/log/maillog
>>> cron.* /var/log/cron
>>> *.emerg :omusrmsg:*
>>> uucp,news.crit /var/log/spooler
>>> local7.* /var/log/boot.log
>>>
>>> *Scott Slattery*
>>>
>>> *Sr. Systems & Cloud Architect*
>>>
>>> *Cloud, Compute, Information & Architecture Team*
>>>
>>> motorolasolutions.com
>>>
>>> *O: 602.529.8226*
>>>
>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>
>>>
>>>
>>>
>>> On Thu, Mar 25, 2021 at 3:20 PM David Lang <david@lang.hm> wrote:
>>>
>>>> what version of rsyslog are you running. can you post your full config?
>>>>
>>>> if you are receiving via TCP and it's not splitting the logs based on
>>>> newlines,
>>>> something very odd is happening.
>>>>
>>>> David Lang
>>>>
>>>> On Thu, 25 Mar 2021, Scott Slattery wrote:
>>>>
>>>> > Date: Thu, 25 Mar 2021 15:07:22 -0700
>>>> > From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>>> > To: David Lang <david@lang.hm>
>>>> > Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
>>>> > rsyslog-users <rsyslog@lists.adiscon.com>,
>>>> mariusz.kruk@safecomp.com
>>>> > Subject: Re: [rsyslog] Altering forwarded logfile names
>>>> >
>>>> > So I'm looking at the pcap and on the first log record the header
>>>> > information looks fine and the record terminated by 0x0A line feed.
>>>> >
>>>> > When I look at the record containing this record written to the
>>>> rsyslog
>>>> > file, the record length is 65586 (64k) long which is what I think you
>>>> were
>>>> > referring to earlier. Why would rsyslog write such a long record? I
>>>> > initially thought my data may be getting truncated so I updated
>>>> > '$MaxMessageSize 64k'. Any suggestion how I fix this behavior.
>>>> >
>>>> > I can confirm that each log record in the pcap is terminated by 0x0A
>>>> and
>>>> > looks in tact so must be something I need to do on rsyslog to fix
>>>> this.
>>>> >
>>>> > *Scott Slattery*
>>>> >
>>>> > *Sr. Systems & Cloud Architect*
>>>> >
>>>> > *Cloud, Compute, Information & Architecture Team*
>>>> >
>>>> > motorolasolutions.com
>>>> >
>>>> > *O: 602.529.8226*
>>>> >
>>>> > *E*: Scott.Slattery@MotorolaSolutions.com
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Thu, Mar 25, 2021 at 1:15 PM David Lang <david@lang.hm> wrote:
>>>> >
>>>> >> you may want to capture with -X so that it decodes it into hex and
>>>> you can
>>>> >> see
>>>> >> newlines vs linefeeds
>>>> >>
>>>> >> David Lang
>>>> >>
>>>> >> On Thu, 25 Mar 2021, Scott Slattery wrote:
>>>> >>
>>>> >>> Date: Thu, 25 Mar 2021 12:47:43 -0700
>>>> >>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>>> >>> To: David Lang <david@lang.hm>
>>>> >>> Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>,
>>>> >>> rsyslog-users <rsyslog@lists.adiscon.com>,
>>>> mariusz.kruk@safecomp.com
>>>> >>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>> >>>
>>>> >>> Thanks David, I just grabbed my first capture and am looking at it.
>>>> I'm
>>>> >>> using TCP vs UDP but your comment makes sense. I'll certainly share
>>>> with
>>>> >>> you my observations once I have some data to refer to. This has
>>>> been a
>>>> >>> learning experience. It's also the first time I've ever seen such an
>>>> >> issue
>>>> >>> with rsyslog so it's a bit of a puzzle.
>>>> >>>
>>>> >>> *Scott Slattery*
>>>> >>>
>>>> >>> *Sr. Systems & Cloud Architect*
>>>> >>>
>>>> >>> *Cloud, Compute, Information & Architecture Team*
>>>> >>>
>>>> >>> motorolasolutions.com
>>>> >>>
>>>> >>> *O: 602.529.8226*
>>>> >>>
>>>> >>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> On Thu, Mar 25, 2021 at 12:32 PM David Lang <david@lang.hm> wrote:
>>>> >>>
>>>> >>>> the rawmsg field in the debugformat output shows exactly what
>>>> rsyslog is
>>>> >>>> seeing.
>>>> >>>>
>>>> >>>> the reason I asked you to check multiple entries is that if
>>>> rsyslog does
>>>> >>>> not see
>>>> >>>> the separator (due to either multiple messages in one UDP packet,
>>>> or
>>>> >>>> missing
>>>> >>>> newlines in a TCP stream) it will combine what are intended as
>>>> multiple
>>>> >>>> messages
>>>> >>>> together into one message as far as rsyslog is concerned, and only
>>>> break
>>>> >>>> it into
>>>> >>>> multiple messages when it hits the configured maxmessagesize size.
>>>> That
>>>> >>>> breaks
>>>> >>>> things at arbitrary points in the message, so as you look at
>>>> multiple
>>>> >>>> messages,
>>>> >>>> they will not all start the same way.
>>>> >>>>
>>>> >>>> a tcpdump -s0 -A -n (with the appropriate port/ip filter) will
>>>> also show
>>>> >>>> exactly
>>>> >>>> what's happening over the wire.
>>>> >>>>
>>>> >>>> David Lang
>>>> >>>>
>>>> >>>> On Thu, 25 Mar 2021, Scott Slattery wrote:
>>>> >>>>
>>>> >>>>> Date: Thu, 25 Mar 2021 12:23:03 -0700
>>>> >>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>>> >>>>> To: Rainer Gerhards <rgerhards@hq.adiscon.com>
>>>> >>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
>>>> >> david@lang.hm
>>>> >>>>> ,
>>>> >>>>> mariusz.kruk@safecomp.com
>>>> >>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>> >>>>>
>>>> >>>>> Hey Guys, before I consume more of your time, I'm capturing the
>>>> packets
>>>> >>>>> from the source device to determine if any of the payload is lost
>>>> on
>>>> >> its
>>>> >>>>> way to the rsyslog collector. After I analyze further, I'll share
>>>> my
>>>> >>>>> observations. I'm hoping nothing is getting dropped over the wire.
>>>> >> Thanks
>>>> >>>>> for your patience and help so far. I expect to have additional
>>>> details
>>>> >>>>> shortly.
>>>> >>>>>
>>>> >>>>> *Scott Slattery*
>>>> >>>>>
>>>> >>>>> *Sr. Systems & Cloud Architect*
>>>> >>>>>
>>>> >>>>> *Cloud, Compute, Information & Architecture Team*
>>>> >>>>>
>>>> >>>>> motorolasolutions.com
>>>> >>>>>
>>>> >>>>> *O: 602.529.8226*
>>>> >>>>>
>>>> >>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On Thu, Mar 25, 2021 at 12:51 AM Rainer Gerhards <
>>>> >>>> rgerhards@hq.adiscon.com>
>>>> >>>>> wrote:
>>>> >>>>>
>>>> >>>>>> Sorry if I didn't catch it, but did you post the configuration?
>>>> If
>>>> >>>>>> not, it would probably be useful.
>>>> >>>>>>
>>>> >>>>>> Looking at the message properties, this looks like the message
>>>> was
>>>> >>>>>> received via UDP but had a LF inside it which was intended as a
>>>> frame
>>>> >>>>>> separator. If so, that is another bug in the vendor's
>>>> implementation.
>>>> >>>>>> Framing is used for TCP, but not UDP. For UDP it is strictly one
>>>> >>>>>> message per package.
>>>> >>>>>>
>>>> >>>>>> HTH,
>>>> >>>>>> Rainer
>>>> >>>>>>
>>>> >>>>>> El mié, 24 mar 2021 a las 22:23, Scott Slattery via rsyslog
>>>> >>>>>> (<rsyslog@lists.adiscon.com>) escribió:
>>>> >>>>>>>
>>>> >>>>>>> From what I can discern since there are lots of transactions,
>>>> there
>>>> >>>> seem
>>>> >>>>>> to
>>>> >>>>>>> be multiple payload formats. For all that I have so far
>>>> analyzed, the
>>>> >>>>>> date
>>>> >>>>>>> truncation issue is the same regardless of the payload
>>>> received. Like
>>>> >>>>>> you,
>>>> >>>>>>> I've never used mmnormalize but I will look at it as an option.
>>>> I've
>>>> >>>>>>> reached out to the product support team to better understand if
>>>> there
>>>> >>>> is
>>>> >>>>>> a
>>>> >>>>>>> appliance-side misconfiguration and something that can be
>>>> corrected
>>>> >>>> there
>>>> >>>>>>> or if I need to make adjustments on the rsyslog side to clean
>>>> up the
>>>> >>>>>>> records.
>>>> >>>>>>>
>>>> >>>>>>> What I meant is that there are multiple message formats (payload
>>>> >> data)
>>>> >>>>>> but
>>>> >>>>>>> there seems to be a consistent truncation of the date field
>>>> just as
>>>> >> you
>>>> >>>>>> saw
>>>> >>>>>>> in the app-name tag field.
>>>> >>>>>>>
>>>> >>>>>>> I appreciate your ongoing feedback and help. This issue is new
>>>> to me
>>>> >>>> and
>>>> >>>>>>> will need a creative solution if the vendor has no answers.
>>>> >>>>>>>
>>>> >>>>>>> Many thanks,
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>> *Scott Slattery*
>>>> >>>>>>>
>>>> >>>>>>> *Sr. Systems & Cloud Architect*
>>>> >>>>>>>
>>>> >>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>> >>>>>>>
>>>> >>>>>>> motorolasolutions.com
>>>> >>>>>>>
>>>> >>>>>>> *O: 602.529.8226*
>>>> >>>>>>>
>>>> >>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>> On Wed, Mar 24, 2021 at 2:14 PM David Lang <david@lang.hm>
>>>> wrote:
>>>> >>>>>>>
>>>> >>>>>>>> when you say 'each is incorrect but the same format'
>>>> >>>>>>>>
>>>> >>>>>>>> does that mean that every log has the year missing? or that
>>>> every
>>>> >> log
>>>> >>>>>> is
>>>> >>>>>>>> combining the logs together?
>>>> >>>>>>>>
>>>> >>>>>>>> I'll note that it's possible to define a custom parser using
>>>> the
>>>> >>>>>>>> mmnormalize
>>>> >>>>>>>> library and add it to the parsing stack. I helped define the
>>>> need
>>>> >> for
>>>> >>>>>> such
>>>> >>>>>>>> a
>>>> >>>>>>>> capability, but haven't used it yet.
>>>> >>>>>>>>
>>>> >>>>>>>> David Lang
>>>> >>>>>>>>
>>>> >>>>>>>> On Wed, 24 Mar 2021, Scott Slattery wrote:
>>>> >>>>>>>>
>>>> >>>>>>>>> Date: Wed, 24 Mar 2021 08:58:07 -0700
>>>> >>>>>>>>> From: Scott Slattery <scott.slattery@motorolasolutions.com>
>>>> >>>>>>>>> To: mariusz.kruk@safecomp.com
>>>> >>>>>>>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <
>>>> >>>>>> david@lang.hm
>>>> >>>>>>>>>
>>>> >>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>> >>>>>>>>>
>>>> >>>>>>>>> Yes, all the logs are coming in this way. There are three
>>>> separate
>>>> >>>>>> logs
>>>> >>>>>>>>> being sent and each is incorrect in its own right but the same
>>>> >> format
>>>> >>>>>>>>> behavior is consistent across each of the three logs. I will
>>>> look
>>>> >>>>>> further
>>>> >>>>>>>>> into the suggestions you have provided and have already
>>>> reached out
>>>> >>>>>> to
>>>> >>>>>>>> the
>>>> >>>>>>>>> appliance support team and will raise the nonconformance with
>>>> the
>>>> >>>>>> RFC.
>>>> >>>>>>>>>
>>>> >>>>>>>>> This does provide me some valuable guidance and I thank each
>>>> of you
>>>> >>>>>> for
>>>> >>>>>>>>> your feedback.
>>>> >>>>>>>>>
>>>> >>>>>>>>> *Scott Slattery*
>>>> >>>>>>>>>
>>>> >>>>>>>>> *Sr. Systems & Cloud Architect*
>>>> >>>>>>>>>
>>>> >>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>> >>>>>>>>>
>>>> >>>>>>>>> motorolasolutions.com
>>>> >>>>>>>>>
>>>> >>>>>>>>> *O: 602.529.8226*
>>>> >>>>>>>>>
>>>> >>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>> >>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>> On Tue, Mar 23, 2021 at 11:22 PM <mariusz.kruk@safecomp.com>
>>>> >> wrote:
>>>> >>>>>>>>>
>>>> >>>>>>>>>> There's something strange with this log entry. Notice that it
>>>> >> looks
>>>> >>>>>> as
>>>> >>>>>>>> if
>>>> >>>>>>>>>> it were more than one log entries joined with LF character
>>>> >> (escaped
>>>> >>>>>> on
>>>> >>>>>>>>>> input as #012). And all "further" entries start with a -
>>>> still
>>>> >>>>>>>>>> nonconformant to the RFC but "fuller" - timestamp containing
>>>> >>>>>> year.It's
>>>> >>>>>>>> the
>>>> >>>>>>>>>> leading event that seems to be trimmed in the front somehow.
>>>> Do
>>>> >> all
>>>> >>>>>>>> events
>>>> >>>>>>>>>> arrive like this?Wys?ano z telefonu Samsung<div>
>>>> >>>>>>>>>> </div><div>
>>>> >>>>>>>>>> </div><div>-------- Original message
>>>> >>>>>>>>>> --------</div><div>From: Scott Slattery via rsyslog <
>>>> >>>>>>>>>> rsyslog@lists.adiscon.com> </div><div>Date: 24/03/2021
>>>> 01:52
>>>> >>>>>>>>>> (GMT+01:00) </div><div>To: David Lang <david@lang.hm>
>>>> >>>>>> </div><div>Cc:
>>>> >>>>>>>>>> Scott Slattery <scott.slattery@motorolasolutions.com>, Scott
>>>> >>>>>> Slattery
>>>> >>>>>>>> via
>>>> >>>>>>>>>> rsyslog <rsyslog@lists.adiscon.com> </div><div>Subject: Re:
>>>> >>>>>> [rsyslog]
>>>> >>>>>>>>>> Altering forwarded logfile names </div><div>
>>>> >>>>>>>>>> </div>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> Hi David, fortunately I had already done this. I'm including
>>>> an
>>>> >>>>>> actual
>>>> >>>>>>>> log
>>>> >>>>>>>>>> entry but have anonymized the data to keep the actual user
>>>> and
>>>> >> email
>>>> >>>>>>>>>> address confidential:
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> Debug line with all properties:
>>>> >>>>>>>>>> FROMHOST: 'ause1oagatst02.aws.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__ause1oagatst02.aws.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=mhxpbOHYU7dKswSuVXqY3cxNk8OtxvgYI4q3_FbE4bU&e=>
>>>> '
>>>> >>>>>>>>>> , fromhost-ip: '10.41.102.143'
>>>> >>>>>>>>>> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__ause1oagatst02.aws.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=mhxpbOHYU7dKswSuVXqY3cxNk8OtxvgYI4q3_FbE4bU&e=>
>>>> '
>>>> >>>>>>>>>> , PRI: 13,
>>>> >>>>>>>>>> syslogtag '03-23T16:'
>>>> >>>>>>>>>> , programname: '03-23T16'
>>>> >>>>>>>>>> , APP-NAME: '03-23T16'
>>>> >>>>>>>>>> , PROCID: '-'
>>>> >>>>>>>>>> , MSGID: '-'
>>>> >>>>>>>>>> , TIMNESTAMP: 'Mar 23 21:47:13'
>>>> >>>>>>>>>> , STRUCTURED-DATA: '-'
>>>> >>>>>>>>>> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__AUSE1OAGATST02.aws.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=C1t9kSGm0Odm-e3_buOIydnR0X69qVM2FVDlfnEKmKg&e=>
>>>> >>>>>>>> ACCESS_GATEWAY
>>>> >>>>>>>>>> ACCESS AUTHZ SESSION INFO USER_SESSION
>>>> >>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>> >> SUBJECT="
>>>> >>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>> >>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>> >>>>>>>>>> apigniodashboard-uat.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__apigniodashboard-2Duat.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=ZsdzmR6uYbEFnBMA7yln_Us0f80yKwi93ui1pqJS31k&e=>
>>>> "
>>>> >>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>>> >>>>>>>> REMOTE_IP="10.44.65.38"
>>>> >>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>> >>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>> >>>>>>>> Safari/537.36"]
>>>> >>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>> >>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__AUSE1OAGATST02.aws.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=C1t9kSGm0Odm-e3_buOIydnR0X69qVM2FVDlfnEKmKg&e=>
>>>> ACCESS_GATEWAY ACCESS AUTHZ
>>>> >> POLICY
>>>> >>>>>>>> INFO
>>>> >>>>>>>>>> USER_AUTHZ
>>>> >> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>> >>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
>>>> >> RESOURCE="/_dash-dependencies"
>>>> >>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED"
>>>> DURATION="0"
>>>> >>>>>>>> APP="Ignio
>>>> >>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>> >>>>>>>>>> apigniodashboard-uat.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__apigniodashboard-2Duat.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=ZsdzmR6uYbEFnBMA7yln_Us0f80yKwi93ui1pqJS31k&e=>"
>>>> RESULT="ALLOW" REASON="N/A -
>>>> >>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
>>>> >> oag_username=
>>>> >>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>>>> >>>>>>>> apigniodashboard-uat.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__apigniodashboard-2Duat.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=ZsdzmR6uYbEFnBMA7yln_Us0f80yKwi93ui1pqJS31k&e=>
>>>> >>>>>>>>>> UserName=Joe.User@mycompantions.com
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>> >>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh;
>>>> Intel Mac
>>>> >>>>>> OS X
>>>> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>>> >> Chrome/88.0.4324.192
>>>> >>>>>>>>>> Safari/537.36 creationTime=1616536016811
>>>> >> maxInactiveInterval=3600000
>>>> >>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>> >>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
>>>> Intel
>>>> >>>>>> Mac
>>>> >>>>>>>> OS X
>>>> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>>> >> Chrome/88.0.4324.192
>>>> >>>>>>>>>> Safari/537.36"] allow access to
>>>> >>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>>>> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__AUSE1OAGATST02.aws.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=C1t9kSGm0Odm-e3_buOIydnR0X69qVM2FVDlfnEKmKg&e=>
>>>> ACCESS_GATEWAY ACCESS AUTHZ
>>>> >>>>>> SESSION
>>>> >>>>>>>> INFO
>>>> >>>>>>>>>> USER_SESSION
>>>> >>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>> >>>>>>>>>> SUBJECT="Joe.User@mycompan'
>>>> >>>>>>>>>> *escaped msg*: '47:20.708-05:00
>>>> AUSE1OAGATST02.aws.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__AUSE1OAGATST02.aws.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=C1t9kSGm0Odm-e3_buOIydnR0X69qVM2FVDlfnEKmKg&e=>
>>>> >>>>>>>>>> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
>>>> >>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>> >> SUBJECT="
>>>> >>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>> >>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>> >>>>>>>>>> apigniodashboard-uat.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__apigniodashboard-2Duat.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=ZsdzmR6uYbEFnBMA7yln_Us0f80yKwi93ui1pqJS31k&e=>
>>>> "
>>>> >>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>>> >>>>>>>> REMOTE_IP="10.44.65.38"
>>>> >>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>> >>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>> >>>>>>>> Safari/537.36"]
>>>> >>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>> >>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__AUSE1OAGATST02.aws.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=C1t9kSGm0Odm-e3_buOIydnR0X69qVM2FVDlfnEKmKg&e=>
>>>> ACCESS_GATEWAY ACCESS AUTHZ
>>>> >> POLICY
>>>> >>>>>>>> INFO
>>>> >>>>>>>>>> USER_AUTHZ
>>>> >> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>> >>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
>>>> >> RESOURCE="/_dash-dependencies"
>>>> >>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED"
>>>> DURATION="0"
>>>> >>>>>>>> APP="Ignio
>>>> >>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>> >>>>>>>>>> apigniodashboard-uat.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__apigniodashboard-2Duat.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=ZsdzmR6uYbEFnBMA7yln_Us0f80yKwi93ui1pqJS31k&e=>"
>>>> RESULT="ALLOW" REASON="N/A -
>>>> >>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
>>>> >> oag_username=
>>>> >>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>>>> >>>>>>>> apigniodashboard-uat.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__apigniodashboard-2Duat.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=ZsdzmR6uYbEFnBMA7yln_Us0f80yKwi93ui1pqJS31k&e=>
>>>> >>>>>>>>>> UserName=Joe.User@mycompantions.com
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>> >>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh;
>>>> Intel Mac
>>>> >>>>>> OS X
>>>> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>>> >> Chrome/88.0.4324.192
>>>> >>>>>>>>>> Safari/537.36 creationTime=1616536016811
>>>> >> maxInactiveInterval=3600000
>>>> >>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>> >>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
>>>> Intel
>>>> >>>>>> Mac
>>>> >>>>>>>> OS X
>>>> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>>> >> Chrome/88.0.4324.192
>>>> >>>>>>>>>> Safari/537.36"] allow access to
>>>> >>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>>>> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__AUSE1OAGATST02.aws.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=C1t9kSGm0Odm-e3_buOIydnR0X69qVM2FVDlfnEKmKg&e=>
>>>> ACCESS_GATEWAY ACCESS AUTHZ
>>>> >>>>>> SESSION
>>>> >>>>>>>> INFO
>>>> >>>>>>>>>> USER_SESSION
>>>> >>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>> >>>>>>>>>> SUBJECT="Joe.User@mycompan'
>>>> >>>>>>>>>> *rawmsg*: '03-23T16:47:20.708-05:00
>>>> >>>>>> AUSE1OAGATST02.aws.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__AUSE1OAGATST02.aws.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=C1t9kSGm0Odm-e3_buOIydnR0X69qVM2FVDlfnEKmKg&e=>
>>>> >>>>>>>>>> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
>>>> >>>>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>> >> SUBJECT="
>>>> >>>>>>>>>> Joe.User@mycompantions.com" APP="Ignio Uat OAG"
>>>> >>>>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>> >>>>>>>>>> apigniodashboard-uat.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__apigniodashboard-2Duat.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=ZsdzmR6uYbEFnBMA7yln_Us0f80yKwi93ui1pqJS31k&e=>
>>>> "
>>>> >>>>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
>>>> >>>>>>>> REMOTE_IP="10.44.65.38"
>>>> >>>>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
>>>> >>>>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
>>>> >>>>>>>> Safari/537.36"]
>>>> >>>>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
>>>> >>>>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00
>>>> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__AUSE1OAGATST02.aws.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=C1t9kSGm0Odm-e3_buOIydnR0X69qVM2FVDlfnEKmKg&e=>
>>>> ACCESS_GATEWAY ACCESS AUTHZ
>>>> >> POLICY
>>>> >>>>>>>> INFO
>>>> >>>>>>>>>> USER_AUTHZ
>>>> >> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>> >>>>>>>>>> SUBJECT="Joe.User@mycompantions.com"
>>>> >> RESOURCE="/_dash-dependencies"
>>>> >>>>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED"
>>>> DURATION="0"
>>>> >>>>>>>> APP="Ignio
>>>> >>>>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
>>>> >>>>>>>>>> apigniodashboard-uat.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__apigniodashboard-2Duat.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=ZsdzmR6uYbEFnBMA7yln_Us0f80yKwi93ui1pqJS31k&e=>"
>>>> RESULT="ALLOW" REASON="N/A -
>>>> >>>>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0
>>>> >> oag_username=
>>>> >>>>>>>>>> Joe.User@mycompantions.com RelayDomain=
>>>> >>>>>>>> apigniodashboard-uat.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__apigniodashboard-2Duat.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=ZsdzmR6uYbEFnBMA7yln_Us0f80yKwi93ui1pqJS31k&e=>
>>>> >>>>>>>>>> UserName=Joe.User@mycompantions.com
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
>>>> >>>>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh;
>>>> Intel Mac
>>>> >>>>>> OS X
>>>> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>>> >> Chrome/88.0.4324.192
>>>> >>>>>>>>>> Safari/537.36 creationTime=1616536016811
>>>> >> maxInactiveInterval=3600000
>>>> >>>>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
>>>> >>>>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;
>>>> Intel
>>>> >>>>>> Mac
>>>> >>>>>>>> OS X
>>>> >>>>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
>>>> >> Chrome/88.0.4324.192
>>>> >>>>>>>>>> Safari/537.36"] allow access to
>>>> >>>>>>>> resource#0122021-03-23T16:47:20.711-05:00
>>>> >>>>>>>>>> AUSE1OAGATST02.aws.mycompany.com
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__AUSE1OAGATST02.aws.mycompany.com&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=8gaKuenx3uBd5Pug9h3wbRLZzFd9VE2COUh_kXxZquY&s=C1t9kSGm0Odm-e3_buOIydnR0X69qVM2FVDlfnEKmKg&e=>
>>>> ACCESS_GATEWAY ACCESS AUTHZ
>>>> >>>>>> SESSION
>>>> >>>>>>>> INFO
>>>> >>>>>>>>>> USER_SESSION
>>>> >>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
>>>> >>>>>>>>>> SUBJECT="Joe.User@mycompan'
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> By inspecting the rawmsg, I can see that field four
>>>> >>>>>> (space-delimited)
>>>> >>>>>>>>>> indicates this is the ACCESS log. So if I were able to
>>>> extract the
>>>> >>>>>> log
>>>> >>>>>>>>>> identifier from the msg, I could then write all access logs
>>>> to the
>>>> >>>>>> same
>>>> >>>>>>>>>> daily file. There are other formats as well from the same
>>>> device
>>>> >>>>>> but the
>>>> >>>>>>>>>> idea is the same.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> *Scott Slattery*
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> motorolasolutions.com
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> *O: 602.529.8226*
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> On Tue, Mar 23, 2021 at 5:29 PM David Lang <david@lang.hm>
>>>> wrote:
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>> the source logfile name is not included in the payload by
>>>> the
>>>> >>>>>> syslog
>>>> >>>>>>>>>> spec.
>>>> >>>>>>>>>>> It
>>>> >>>>>>>>>>> may be in the case of your appliance, but we would need to
>>>> see a
>>>> >>>>>> sample
>>>> >>>>>>>>>>> log to
>>>> >>>>>>>>>>> understand ho to parse it.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> based on your template, you are using app-name, which may be
>>>> >> listed
>>>> >>>>>>>>>>> separtely if
>>>> >>>>>>>>>>> it's a RFC5424 format log, or may be part of the syslog tag
>>>> if
>>>> >>>>>> it's a
>>>> >>>>>>>>>>> RFC3164
>>>> >>>>>>>>>>> format log over the wire (neither format has a way to
>>>> specify a
>>>> >>>>>> source
>>>> >>>>>>>>>> log
>>>> >>>>>>>>>>> file
>>>> >>>>>>>>>>> by default)
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> you can look at
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
>>>> >>>>>>>>>>> and see the *-cc
>>>> >>>>>>>>>>> options that you could apply to the app-name to eliminate
>>>> control
>>>> >>>>>>>>>>> characters.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Again, we really need to see the original log message to
>>>> >> understand
>>>> >>>>>>>>>> what's
>>>> >>>>>>>>>>> what.
>>>> >>>>>>>>>>> Please log it with the templateRSYSLOG_DebugFormat so we
>>>> can see
>>>> >>>>>>>> exactly
>>>> >>>>>>>>>>> what is
>>>> >>>>>>>>>>> sent over the wire and how rsyslog has parsed it.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> David Lang
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog
>>>> >>>>>>>>>>> wrote:
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>>> Date: Tue, 23 Mar 2021 16:05:45 -0700
>>>> >>>>>>>>>>>> From: Scott Slattery via rsyslog <
>>>> rsyslog@lists.adiscon.com>
>>>> >>>>>>>>>>>> To: John Chivian <jchivian@chivian.com>
>>>> >>>>>>>>>>>> Cc: Scott Slattery <scott.slattery@motorolasolutions.com>,
>>>> >>>>>>>>>>>> rsyslog-users <rsyslog@lists.adiscon.com>
>>>> >>>>>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> Thanks, John, let me try to clarify what I mean.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> Normally when I forward from a remote server to the
>>>> central log
>>>> >>>>>>>>>> server, I
>>>> >>>>>>>>>>>> can include a tag that can then be used to determine the
>>>> file
>>>> >>>>>> name I
>>>> >>>>>>>>>> want
>>>> >>>>>>>>>>>> on the central server. Since I have no real way to include
>>>> this
>>>> >>>>>> tag
>>>> >>>>>>>>>> from
>>>> >>>>>>>>>>>> the appliance, this is not an option.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> I'm looking for a way of inspecting the incoming packets to
>>>> >>>>>>>> determining
>>>> >>>>>>>>>>> the
>>>> >>>>>>>>>>>> source logfile name (which is included in the payload) and
>>>> use
>>>> >>>>>> that
>>>> >>>>>>>>>>>> filename on the target central server. Since there are
>>>> multiple
>>>> >>>>>> logs
>>>> >>>>>>>>>>> being
>>>> >>>>>>>>>>>> sent (access, audit, monitor, etc.), I'd like to segregate
>>>> these
>>>> >>>>>> into
>>>> >>>>>>>>>>> their
>>>> >>>>>>>>>>>> own files. I'm already using a template with the host
>>>> >> information
>>>> >>>>>> to
>>>> >>>>>>>>>>>> dynamically create the file names. I just don't know how I
>>>> can
>>>> >> go
>>>> >>>>>>>>>> beyond
>>>> >>>>>>>>>>>> this to also include the source logname.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> Here's the template I'm using. It works for all other hosts
>>>> >> where
>>>> >>>>>> I
>>>> >>>>>>>> can
>>>> >>>>>>>>>>>> configure the tag but I get garbage names from the
>>>> appliance. I
>>>> >>>>>> had
>>>> >>>>>>>>>> hoped
>>>> >>>>>>>>>>>> that the appliance included some standard syslog tags but
>>>> it
>>>> >>>>>> doesn't
>>>> >>>>>>>>>> seem
>>>> >>>>>>>>>>>> so.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> template(name="DynRemoteLogFile" type="string"
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> *Scott Slattery*
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> motorolasolutions.com
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> *O: 602.529.8226*
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <
>>>> >>>>>> jchivian@chivian.com>
>>>> >>>>>>>>>>> wrote:
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>> Your use of the term “file name” is confusing. When
>>>> senders
>>>> >>>>>> deliver
>>>> >>>>>>>>>> to
>>>> >>>>>>>>>>>>> rsyslog over the network there is no exchange of files or
>>>> >>>>>> filenames,
>>>> >>>>>>>>>>> only
>>>> >>>>>>>>>>>>> packets of information. Those packets are expected to be
>>>> in a
>>>> >>>>>> format
>>>> >>>>>>>>>>> that
>>>> >>>>>>>>>>>>> syslog understands such that useful information (header
>>>> >> elements
>>>> >>>>>> and
>>>> >>>>>>>>>>>>> message body) may be parsed from them. If you as the
>>>> rsyslog
>>>> >>>>>> admin
>>>> >>>>>>>>>>> choose
>>>> >>>>>>>>>>>>> to use some of that header information to compose
>>>> filenames for
>>>> >>>>>>>> output
>>>> >>>>>>>>>>>>> files, then yes you are sort of at the mercy of the
>>>> senders
>>>> >>>>>> content
>>>> >>>>>>>>>>>>> (especially if the sender doesn’t follow the syslog
>>>> rules).
>>>> >>>>>> However,
>>>> >>>>>>>>>>> there
>>>> >>>>>>>>>>>>> are functions in the advanced syntax that can be used to
>>>> >> perform
>>>> >>>>>> the
>>>> >>>>>>>>>>> type
>>>> >>>>>>>>>>>>> of character replacements you’re talking about.
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> It is common practice to use the syslog header/rsyslog
>>>> property
>>>> >>>>>>>>>> element
>>>> >>>>>>>>>>>>> called “hostname” for just such purposes. Is this what
>>>> you’re
>>>> >>>>>>>> talking
>>>> >>>>>>>>>>>>> about? You’d have to provide your configuration for real
>>>> >>>>>> analysis,
>>>> >>>>>>>> at
>>>> >>>>>>>>>>>>> least the part you perceive to be responsible for the
>>>> problem.
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> Regards,
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
>>>> >>>>>>>>>>>>> rsyslog@lists.adiscon.com> wrote:
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> I have a configured central log collector using rsyslog.
>>>> A few
>>>> >>>>>> of
>>>> >>>>>>>>>> the
>>>> >>>>>>>>>>>>>> devices forwarding their logs are appliances that have no
>>>> >>>>>>>>>>> configuration
>>>> >>>>>>>>>>>>>> options other than the IP forwarding address and
>>>> protocol. I
>>>> >>>>>> cannot
>>>> >>>>>>>>>>>>> control
>>>> >>>>>>>>>>>>>> what file names are being sent.
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> Unfortunately, they are sending unintelligible file
>>>> names with
>>>> >>>>>>>>>>> characters
>>>> >>>>>>>>>>>>>> that normally would be escaped. Is there any way I can
>>>> control
>>>> >>>>>> or
>>>> >>>>>>>>>>> alter
>>>> >>>>>>>>>>>>> the
>>>> >>>>>>>>>>>>>> incoming file name to normalize it to avoid these odd
>>>> >>>>>> characters?
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> For example, could I establish a character map that maps
>>>> the
>>>> >>>>>>>>>> unallowed
>>>> >>>>>>>>>>>>>> character to something acceptable?
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> thanks,
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> *Scott Slattery*
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> *Sr. Systems & Cloud Architect*
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team*
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> motorolasolutions.com
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> *O: 602.529.822*
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> *E*: Scott.Slattery@MotorolaSolutions.com
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> --
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> *For more information on how and why we collect your
>>>> personal
>>>> >>>>>>>>>>>>>> information, please visit our Privacy Policy
>>>> >>>>>>>>>>>>>> <
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>> >>>>>>>>>>>>>> .*
>>>> >>>>>>>>>>>>>> _______________________________________________
>>>> >>>>>>>>>>>>>> rsyslog mailing list
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
>>>> >>>>>>>>>>>>>> What's up with rsyslog? Follow
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
>>>> >>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
>>>> ARCHIVED
>>>> >> by
>>>> >>>>>> a
>>>> >>>>>>>>>>> myriad
>>>> >>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO
>>>> NOT POST
>>>> >>>>>> if
>>>> >>>>>>>> you
>>>> >>>>>>>>>>>>> DON'T LIKE THAT.
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> --
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> *For more information on how and why we collect your
>>>> personal
>>>> >>>>>>>>>>>> information, please visit our Privacy Policy
>>>> >>>>>>>>>>>> <
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>> >>>>>>>>>>>> .*
>>>> >>>>>>>>>>>> _______________________________________________
>>>> >>>>>>>>>>>> rsyslog mailing list
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
>>>> >>>>>>>>>>>> What's up with rsyslog? Follow
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
>>>> >>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
>>>> ARCHIVED by
>>>> >> a
>>>> >>>>>>>>>> myriad
>>>> >>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>> POST
>>>> >> if
>>>> >>>>>> you
>>>> >>>>>>>>>>> DON'T LIKE THAT.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> --
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> *For more information on how and why we collect your personal
>>>> >>>>>>>>>> information, please visit our Privacy Policy
>>>> >>>>>>>>>> <
>>>> >>>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>> >>>>>>>>>>> .*
>>>> >>>>>>>>>> _______________________________________________
>>>> >>>>>>>>>> rsyslog mailing list
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
>>>> >>>>>>>>>> What's up with rsyslog? Follow
>>>> >>>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
>>>> >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>> by a
>>>> >>>>>> myriad
>>>> >>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>> POST if
>>>> >>>>>> you
>>>> >>>>>>>>>> DON'T LIKE THAT.
>>>> >>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>
>>>> >>>>>>> --
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>> *For more information on how and why we collect your personal
>>>> >>>>>>> information, please visit our Privacy Policy
>>>> >>>>>>> <
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>>>> >>>>>>> .*
>>>> >>>>>>> _______________________________________________
>>>> >>>>>>> rsyslog mailing list
>>>> >>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=7hc9b2gSfvsfQzXvsRVXqGeP4Rz_tJtrykpjqkGBIdk&e=
>>>> >>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=bDybhvts9KqI-WEfMHo7O40ulQK5M6Tg4tnKNRRIhE8&e=
>>>> >>>>>>> What's up with rsyslog? Follow
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=zPhpcUPlK_CWLiU75_WwjyQGz5mFTiXc1yyBIfppkrI&e=
>>>> >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>>>> a
>>>> >>>> myriad
>>>> >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>>> if you
>>>> >>>>>> DON'T LIKE THAT.
>>>> >>>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>
>>>> >>>
>>>> >
>>>> >
>>>
>>>
>> *For more information on how and why we collect your personal
>> information, please visit our Privacy Policy
>> <https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
>>
>

--

*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.