Ok gents, here you go This is rsyslog v8.2102…
## Start rsyslog.conf
$AbortOnUncleanConfig on
$RepeatedMsgReduction off
$umask 0000
$CreateDirs on
global(
processInternalMessages="on"
parser.dropTrailingLFOnReception="on"
parser.spaceLFOnReceive="on"
parser.escapeControlCharactersOnReceive="on"
preserveFQDN="on"
net.enableDNS="off"
workDirectory="/home/user/demo"
maxMessageSize="16k"
)
module(load="imudp")
module(load="imptcp")
module(load="imfile")
module(load="mmjsonparse")
module(load="builtin:omfile" Template="RSYSLOG_FileFormat")
template(name="demo_out" type="list") {
constant(value="R: ") property(name="rawmsg") constant(value="\n")
constant(value="M: ") property(name="msg") constant(value="\n")
constant(value="I: ") property(name="inputname") constant(value="\n")
constant(value="S: ") property(name="$!strmsg" format="jsonr") constant(value="\n")
constant(value="J: ") property(name="$!jsonmsg") constant(value="\n\n")
}
ruleset(name="json_check") {
set $.isjson = parse_json($msg, "\$!parsed");
if ($.isjson == 0) then {
set $!jsonmsg = $msg;
set $!strmsg = "";
} else {
set $!jsonmsg = "null";
set $!strmsg = $msg;
}
}
ruleset(name="to_debug") {
action(
type="omfile"
name="debug_demo"
template="demo_out"
file="/home/user/demo/debug.out"
)
}
ruleset(name="from_Demo") {
call json_check
call to_debug
}
input(type="imudp" name="udp-55141-in" port="55141" ruleset="from_Demo")
input(type="imptcp" name="ptcp-55141-in" port="55141" ruleset="from_Demo")
input(type="imfile"
file="/home/user/practice/*.demolog"
tag="Demo"
facility="auth"
severity="info"
persiststateinterval="0"
readmode="0"
readtimeout="5"
ruleset="from_Demo"
)
## End rsyslog.conf
Then, from the output file, some correct examples...
R: <111>1 2021-06-30T15:05:48.323+00:00 azlapncismlog02 logaudit 22815 test - test message at 1625065548.325
M: test message at 1625065548.325
I: ptcp-55141-in
S: test message at 1625065548.325
J: null
R: <111>1 2021-06-30T15:05:55.911+00:00 azlapncismlog02 logaudit 25337 test - test message at 1625065555.912
M: test message at 1625065555.912
I: udp-55141-in
S: test message at 1625065555.912
J: null
R: test message at 1625065662.605
M: test message at 1625065662.605
I: imfile
S: test message at 1625065662.605
J: null
R: a b c d e f
M: a b c d e f
I: imfile
S: a b c d e f
J: null
R: a,b,c,d,e,f
M: a,b,c,d,e,f
I: imfile
S: a,b,c,d,e,f
J: null
R: a|b|c|d|e|f
M: a|b|c|d|e|f
I: imfile
S: a|b|c|d|e|f
J: null
R: <111>1 2021-06-30T15:13:24.190+00:00 azlapncismlog02 logaudit 4266 test - {"str_field":"pi","num_field":3.14159}
M: {"str_field":"pi","num_field":3.14159}
I: ptcp-55141-in
S:
J: {"str_field":"pi","num_field":3.14159}
Then, from the output file, some incorrect examples...
R: 0,1,2,3,4,5
M: 0,1,2,3,4,5
I: imfile
S:
J: 0,1,2,3,4,5
R: <111>1 2021-06-30T15:13:51.909+00:00 azlapncismlog02 logaudit 4981 test - "str_field":"pi","num_field":3.14159
M: "str_field":"pi","num_field":3.14159
I: ptcp-55141-in
S:
J: "str_field":"pi","num_field":3.14159
R: <111>1 2021-06-30T15:15:56.392+00:00 azlapncismlog02 logaudit 8469 test - 003491626A431T|Success|2021-04-28 22:15:31|event.type.sybtype|10.10.10.10
M: 003491626A431T|Success|2021-04-28 22:15:31|event.type.sybtype|10.10.10.10
I: ptcp-55141-in
S:
J: 003491626A431T|Success|2021-04-28 22:15:31|event.type.sybtype|10.10.10.10
> On Jun 17, 2021, at 12:03, David Lang <david@lang.hm> wrote:
>
> what is msg and rawmsg in these two cases?
>
> what you are showing is the output, what we need to look at is the input to be parsed.
>
> David Lang
>
> On Thu, 17 Jun 2021, John Chivian wrote:
>
>> Date: Thu, 17 Jun 2021 11:37:14 -0500
>> From: John Chivian <jchivian@chivian.com>
>> To: David Lang <david@lang.hm>
>> Cc: John Chivian via rsyslog <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] Odd parse_json Behavior With imfile Input
>> Hi David:
>>
>> I wish I could say that was the case, but the msg object is exactly what it should be. Here’s the output in error...
>>
>> "string": "", "json": 206|domain1\user1|SEC_ACCOUNTING_CONSULTANT|N|2019-10-21 15:46:33.487|domain2\user2|10.224.129.254
>>
>> …and here’s the output with parentheses added…
>>
>> "string": "(206|domain1\\user1|SEC_ACCOUNTING_CONSULTANT|N|2019-10-21 15:46:33.487|domain2\\user2|10.224.129.254)", "json": "null"
>>
>> …produced by…
>>
>> constant(value="\"string\": \"") property(name="$!smsg" format="jsonr") constant(value="\",")
>> constant(value="\"json\": ") property(name="$!jmsg")
>> As you can see the only difference in msg is the added parenthesis, and both represent exactly what msg should be.
>>
>>
>> Regards,
>>
>>
>>
>>> On Jun 17, 2021, at 11:16, David Lang <david@lang.hm> wrote:
>>>
>>> whenever you have a parsing problem, log the messages with the template RSYSLOG_DebugFormat
>>>
>>> 90%+ of the time, we find that the contents of the variables are not what you are expecting, and the rest of the time, seeing the debug output lets us figure out what is going wrong with the parser
>>>
>>> Also, what version of rsyslog are you running?
>>>
>>> David Lang
>>>
>>> On Thu, 17 Jun 2021, John Chivian via rsyslog wrote:
>>>
>>>> Date: Thu, 17 Jun 2021 11:05:21 -0500
>>>> From: John Chivian via rsyslog <rsyslog@lists.adiscon.com>
>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Cc: John Chivian <jchivian@chivian.com>
>>>> Subject: [rsyslog] Odd parse_json Behavior With imfile Input
>>>> I am using the parse_json function to determine whether or not the msg element is a JSON object.
>>>>
>>>> set $.isjson = parse_json($msg, "\$!parsed");
>>>>
>>>> if ($.isjson == 0) then {
>>>> set $!jmsg = $msg;
>>>> set $!smsg = "";
>>>> } else {
>>>> set $!jmsg = "null";
>>>> set $!smsg = $msg;
>>>> }
>>>>
>>>>
>>>> For network input this seems to always work, but for file input we seeing odd behavior where some strings that are not JSON objects are being identified as if they were JSON objects.
>>>>
>>>> Case in point is the msg "a,b,c,d,e,f”. If received over the network (imptcp) with a proper syslog header, this msg is correctly identified as not being a JSON object, and the value is applied to the $!smsg element in the if construct that follows.
>>>>
>>>> However, if that same string is received in file input (imfile), the message is incorrectly identified as being a JSON object and the value is applied to the $!jmsg element. The same is true if the fields are pipe delimited as "a|b|c|d|e|f".
>>>> If I modify the msg to be wrapped in parenthesis, i.e. "(a,b,c,d,e,f)” or “(a|b|c|d|e|f)", then the string is identified properly in both cases.
>>>>
>>>> Another thing that may be of significance is that if I add a syslog header to each line of the file input, and specify needparse=“on” in the file input definition, the issue goes away and the msg element is always identified properly.
>>>>
>>>> This is v8.2102. I’m not sure if this is a bug or if I’m missing something fundamental about the differences between input types, so any advice or guidance is greatly appreciated.
>>>>
>>>> Thanks all!
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
## Start rsyslog.conf
$AbortOnUncleanConfig on
$RepeatedMsgReduction off
$umask 0000
$CreateDirs on
global(
processInternalMessages="on"
parser.dropTrailingLFOnReception="on"
parser.spaceLFOnReceive="on"
parser.escapeControlCharactersOnReceive="on"
preserveFQDN="on"
net.enableDNS="off"
workDirectory="/home/user/demo"
maxMessageSize="16k"
)
module(load="imudp")
module(load="imptcp")
module(load="imfile")
module(load="mmjsonparse")
module(load="builtin:omfile" Template="RSYSLOG_FileFormat")
template(name="demo_out" type="list") {
constant(value="R: ") property(name="rawmsg") constant(value="\n")
constant(value="M: ") property(name="msg") constant(value="\n")
constant(value="I: ") property(name="inputname") constant(value="\n")
constant(value="S: ") property(name="$!strmsg" format="jsonr") constant(value="\n")
constant(value="J: ") property(name="$!jsonmsg") constant(value="\n\n")
}
ruleset(name="json_check") {
set $.isjson = parse_json($msg, "\$!parsed");
if ($.isjson == 0) then {
set $!jsonmsg = $msg;
set $!strmsg = "";
} else {
set $!jsonmsg = "null";
set $!strmsg = $msg;
}
}
ruleset(name="to_debug") {
action(
type="omfile"
name="debug_demo"
template="demo_out"
file="/home/user/demo/debug.out"
)
}
ruleset(name="from_Demo") {
call json_check
call to_debug
}
input(type="imudp" name="udp-55141-in" port="55141" ruleset="from_Demo")
input(type="imptcp" name="ptcp-55141-in" port="55141" ruleset="from_Demo")
input(type="imfile"
file="/home/user/practice/*.demolog"
tag="Demo"
facility="auth"
severity="info"
persiststateinterval="0"
readmode="0"
readtimeout="5"
ruleset="from_Demo"
)
## End rsyslog.conf
Then, from the output file, some correct examples...
R: <111>1 2021-06-30T15:05:48.323+00:00 azlapncismlog02 logaudit 22815 test - test message at 1625065548.325
M: test message at 1625065548.325
I: ptcp-55141-in
S: test message at 1625065548.325
J: null
R: <111>1 2021-06-30T15:05:55.911+00:00 azlapncismlog02 logaudit 25337 test - test message at 1625065555.912
M: test message at 1625065555.912
I: udp-55141-in
S: test message at 1625065555.912
J: null
R: test message at 1625065662.605
M: test message at 1625065662.605
I: imfile
S: test message at 1625065662.605
J: null
R: a b c d e f
M: a b c d e f
I: imfile
S: a b c d e f
J: null
R: a,b,c,d,e,f
M: a,b,c,d,e,f
I: imfile
S: a,b,c,d,e,f
J: null
R: a|b|c|d|e|f
M: a|b|c|d|e|f
I: imfile
S: a|b|c|d|e|f
J: null
R: <111>1 2021-06-30T15:13:24.190+00:00 azlapncismlog02 logaudit 4266 test - {"str_field":"pi","num_field":3.14159}
M: {"str_field":"pi","num_field":3.14159}
I: ptcp-55141-in
S:
J: {"str_field":"pi","num_field":3.14159}
Then, from the output file, some incorrect examples...
R: 0,1,2,3,4,5
M: 0,1,2,3,4,5
I: imfile
S:
J: 0,1,2,3,4,5
R: <111>1 2021-06-30T15:13:51.909+00:00 azlapncismlog02 logaudit 4981 test - "str_field":"pi","num_field":3.14159
M: "str_field":"pi","num_field":3.14159
I: ptcp-55141-in
S:
J: "str_field":"pi","num_field":3.14159
R: <111>1 2021-06-30T15:15:56.392+00:00 azlapncismlog02 logaudit 8469 test - 003491626A431T|Success|2021-04-28 22:15:31|event.type.sybtype|10.10.10.10
M: 003491626A431T|Success|2021-04-28 22:15:31|event.type.sybtype|10.10.10.10
I: ptcp-55141-in
S:
J: 003491626A431T|Success|2021-04-28 22:15:31|event.type.sybtype|10.10.10.10
> On Jun 17, 2021, at 12:03, David Lang <david@lang.hm> wrote:
>
> what is msg and rawmsg in these two cases?
>
> what you are showing is the output, what we need to look at is the input to be parsed.
>
> David Lang
>
> On Thu, 17 Jun 2021, John Chivian wrote:
>
>> Date: Thu, 17 Jun 2021 11:37:14 -0500
>> From: John Chivian <jchivian@chivian.com>
>> To: David Lang <david@lang.hm>
>> Cc: John Chivian via rsyslog <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] Odd parse_json Behavior With imfile Input
>> Hi David:
>>
>> I wish I could say that was the case, but the msg object is exactly what it should be. Here’s the output in error...
>>
>> "string": "", "json": 206|domain1\user1|SEC_ACCOUNTING_CONSULTANT|N|2019-10-21 15:46:33.487|domain2\user2|10.224.129.254
>>
>> …and here’s the output with parentheses added…
>>
>> "string": "(206|domain1\\user1|SEC_ACCOUNTING_CONSULTANT|N|2019-10-21 15:46:33.487|domain2\\user2|10.224.129.254)", "json": "null"
>>
>> …produced by…
>>
>> constant(value="\"string\": \"") property(name="$!smsg" format="jsonr") constant(value="\",")
>> constant(value="\"json\": ") property(name="$!jmsg")
>> As you can see the only difference in msg is the added parenthesis, and both represent exactly what msg should be.
>>
>>
>> Regards,
>>
>>
>>
>>> On Jun 17, 2021, at 11:16, David Lang <david@lang.hm> wrote:
>>>
>>> whenever you have a parsing problem, log the messages with the template RSYSLOG_DebugFormat
>>>
>>> 90%+ of the time, we find that the contents of the variables are not what you are expecting, and the rest of the time, seeing the debug output lets us figure out what is going wrong with the parser
>>>
>>> Also, what version of rsyslog are you running?
>>>
>>> David Lang
>>>
>>> On Thu, 17 Jun 2021, John Chivian via rsyslog wrote:
>>>
>>>> Date: Thu, 17 Jun 2021 11:05:21 -0500
>>>> From: John Chivian via rsyslog <rsyslog@lists.adiscon.com>
>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Cc: John Chivian <jchivian@chivian.com>
>>>> Subject: [rsyslog] Odd parse_json Behavior With imfile Input
>>>> I am using the parse_json function to determine whether or not the msg element is a JSON object.
>>>>
>>>> set $.isjson = parse_json($msg, "\$!parsed");
>>>>
>>>> if ($.isjson == 0) then {
>>>> set $!jmsg = $msg;
>>>> set $!smsg = "";
>>>> } else {
>>>> set $!jmsg = "null";
>>>> set $!smsg = $msg;
>>>> }
>>>>
>>>>
>>>> For network input this seems to always work, but for file input we seeing odd behavior where some strings that are not JSON objects are being identified as if they were JSON objects.
>>>>
>>>> Case in point is the msg "a,b,c,d,e,f”. If received over the network (imptcp) with a proper syslog header, this msg is correctly identified as not being a JSON object, and the value is applied to the $!smsg element in the if construct that follows.
>>>>
>>>> However, if that same string is received in file input (imfile), the message is incorrectly identified as being a JSON object and the value is applied to the $!jmsg element. The same is true if the fields are pipe delimited as "a|b|c|d|e|f".
>>>> If I modify the msg to be wrapped in parenthesis, i.e. "(a,b,c,d,e,f)” or “(a|b|c|d|e|f)", then the string is identified properly in both cases.
>>>>
>>>> Another thing that may be of significance is that if I add a syslog header to each line of the file input, and specify needparse=“on” in the file input definition, the issue goes away and the msg element is always identified properly.
>>>>
>>>> This is v8.2102. I’m not sure if this is a bug or if I’m missing something fundamental about the differences between input types, so any advice or guidance is greatly appreciated.
>>>>
>>>> Thanks all!
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.