Mailing List Archive

separate pid from syslogtag
list members,

i am looking to separate the PID from the SysLogTag in all messages, and
have separate fields in the LogAnalyser database.  i have several hosts
that send messages via RELP to a central cluster of receivers, and those
cluster members then submit the messages into MariaDB.

as i understand it, i need to:

1) modify each source hosts messages to split out the PID from the
SysLogTag field
2) modify the central clusters inserts to MariaDB, to specify the PID
independent of the SysLogTag
3) modify the view used in LogAnalyzer to show the PID column

for the first bullet, my typical syslog source has the following config:

global(workDirectory="/var/lib/rsyslog")
module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")

module(load="imuxsock"
       SysSock.Use="off")

module(load="imjournal"
       StateFile="imjournal.state")

include(file="/etc/rsyslog.d/*.conf" mode="optional")

module(load="omrelp") # needs to be done just once

action(type="omrelp" target="relp.bpk2.com" port="20514")

&~

i believe i need to create my own Template, and specify that instead of
"RSYSLOG_TraditionalFileFormat". Being that the source hosts do not
directly insert the messages into the database, i dont need to specify
an insert at this point.  i believe RSYSLOG_TraditionalFileFormat has
the following fields:

$template TraditionalFileFormat,"%TIMESTAMP% %HOSTNAME%
%syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"

what fields do i need to change, to use the raw SysLogTag, and PID as
separate fields to send to the central receivers?

for the second bullet, i insert messages to the database using the
following config:

global(workDirectory="/var/lib/rsyslog")

module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")

include(file="/etc/rsyslog.d/*.conf" mode="optional")

module(load="imuxsock"
       SysSock.Use="off")

module(load="imjournal"
       StateFile="imjournal.state")

module(load="imudp")
input(type="imudp" port="514")

module(load="imtcp")
input(type="imtcp" port="514")

module(load="imrelp")
input(type="imrelp" port="20514")

module(load="ommysql")
action(type="ommysql" server="database.domain.tld" serverport="3306"
db="Syslog" uid="" pwd="")

&~

i figure i need to specify another template here, and use that for the
inserts.  i found the following, and want to sanity check it, as i dont
know alot about RegEx, which seems to be used:

$template dbFormat,"insert into SystemEvents (Message, Facility,
FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID,
SysLogTag, processid) values ('%msg%', %syslogfacility%,
'%HOSTNAME%',%syslogpriority%, '%timereported:::date-mysql%',
'%timegenerated:::date-mysql%', %iut%,
'%syslogtag:R,ERE,1,FIELD:(.+)(\[[0-9]{1,10}\]).*--end%',
'%syslogtag:R,ERE,1,BLANK:\[([0-9]{1,10})\]--end%')",sql

with the above Template, called dbFormat, i would modify the "action" of
my ommysql module, to be:

...
action(type="ommysql" server="database.domain.tld" serverport="3306"
db="Syslog" uid="" pwd="" template="dbFormat")

am i tracking properly on this?  the database does have the processid
field defined already, so it seems that i just need to populate that field.

for the third bullet, i have installed the helper pieces for the
phpLogCon/LogAnalyzer, so i just need to update the configs in the
database to show the ProcessID field in the view i defined and set as
the default.  again, am i thinking about this in the right terms?  are
there any steps i am overlooking?

thanks in advance,

brendan kearney
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: separate pid from syslogtag [ In reply to ]
list members,

i never received responses from the below, and i looked in my junk mail
folders, so i am replying to my own message.

on the central cluster servers, i made a couple changes.  I added a
template for the SQL format, and specified that format in the ommysql
action stanza...

# Define the MariaDB/SQL insert template
$template bpk2_SqlWithSeparatePID,"insert into SystemEvents (Message,
Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
InfoUnitID, SysLogTag, processid) values ('%msg%', %syslogfacility%,
'%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%',
'%timegenerated:::date-mysql%', %iut%, '%programname%', '%PROCID%')",SQL

# Provides MySQL syslog transmission
# for parameters see http://www.rsyslog.com/doc/ommysql.html
module(load="ommysql") # needs to be done just once
action(type="ommysql" server="database.bpk2.com" serverport="3306"
       db="Syslog" uid="Syslog" pwd="SooperSekretString"
template="bpk2_SqlWithSeparatePID")

i now get the SysLogTag name without the PID, from those cluster
servers, but i need to have the client's data parsed in the same way.  i
tried creating a format on the client and specifying the format in the
omrelp stanza, but that does not seem to be everything i need to do, if
it will even work.

i need to figure out how to have the clients syslogtag parsed, to
separate the "programname" and "procid" strings.  this needs to be done,
seemingly on the server with a property replacer, and then mapped to the
fields used in the database template, so they are inserted properly into
the database.  are there any pointers on how this is best/properly done?

thank you,

brendan

On 2/25/22 10:52 AM, Brendan Kearney wrote:
>
> list members,
>
> i am looking to separate the PID from the SysLogTag in all messages,
> and have separate fields in the LogAnalyser database. i have several
> hosts that send messages via RELP to a central cluster of receivers,
> and those cluster members then submit the messages into MariaDB.
>
> as i understand it, i need to:
>
> 1) modify each source hosts messages to split out the PID from the
> SysLogTag field
> 2) modify the central clusters inserts to MariaDB, to specify the PID
> independent of the SysLogTag
> 3) modify the view used in LogAnalyzer to show the PID column
>
> for the first bullet, my typical syslog source has the following config:
>
> global(workDirectory="/var/lib/rsyslog")
> module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")
>
> module(load="imuxsock"
>        SysSock.Use="off")
>
> module(load="imjournal"
>        StateFile="imjournal.state")
>
> include(file="/etc/rsyslog.d/*.conf" mode="optional")
>
> module(load="omrelp") # needs to be done just once
>
> action(type="omrelp" target="relp.bpk2.com" port="20514")
>
> &~
>
> i believe i need to create my own Template, and specify that instead
> of "RSYSLOG_TraditionalFileFormat". Being that the source hosts do not
> directly insert the messages into the database, i dont need to specify
> an insert at this point.  i believe RSYSLOG_TraditionalFileFormat has
> the following fields:
>
> $template TraditionalFileFormat,"%TIMESTAMP% %HOSTNAME%
> %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
>
> what fields do i need to change, to use the raw SysLogTag, and PID as
> separate fields to send to the central receivers?
>
> for the second bullet, i insert messages to the database using the
> following config:
>
> global(workDirectory="/var/lib/rsyslog")
>
> module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")
>
> include(file="/etc/rsyslog.d/*.conf" mode="optional")
>
> module(load="imuxsock"
>        SysSock.Use="off")
>
> module(load="imjournal"
>        StateFile="imjournal.state")
>
> module(load="imudp")
> input(type="imudp" port="514")
>
> module(load="imtcp")
> input(type="imtcp" port="514")
>
> module(load="imrelp")
> input(type="imrelp" port="20514")
>
> module(load="ommysql")
> action(type="ommysql" server="database.domain.tld"
> serverport="3306" db="Syslog" uid="" pwd="")
>
> &~
>
> i figure i need to specify another template here, and use that for the
> inserts.  i found the following, and want to sanity check it, as i
> dont know alot about RegEx, which seems to be used:
>
> $template dbFormat,"insert into SystemEvents (Message, Facility,
> FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID,
> SysLogTag, processid) values ('%msg%', %syslogfacility%,
> '%HOSTNAME%',%syslogpriority%, '%timereported:::date-mysql%',
> '%timegenerated:::date-mysql%', %iut%,
> '%syslogtag:R,ERE,1,FIELD:(.+)(\[[0-9]{1,10}\]).*--end%',
> '%syslogtag:R,ERE,1,BLANK:\[([0-9]{1,10})\]--end%')",sql
>
> with the above Template, called dbFormat, i would modify the "action"
> of my ommysql module, to be:
>
> ...
> action(type="ommysql" server="database.domain.tld"
> serverport="3306" db="Syslog" uid="" pwd="" template="dbFormat")
>
> am i tracking properly on this?  the database does have the processid
> field defined already, so it seems that i just need to populate that
> field.
>
> for the third bullet, i have installed the helper pieces for the
> phpLogCon/LogAnalyzer, so i just need to update the configs in the
> database to show the ProcessID field in the view i defined and set as
> the default.  again, am i thinking about this in the right terms?  are
> there any steps i am overlooking?
>
> thanks in advance,
>
> brendan kearney
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.