Mailing List Archive

Odd parse_json Behavior With imfile Input
I am using the parse_json function to determine whether or not the msg element is a JSON object.

set $.isjson = parse_json($msg, "\$!parsed");

if ($.isjson == 0) then {
set $!jmsg = $msg;
set $!smsg = "";
} else {
set $!jmsg = "null";
set $!smsg = $msg;
}


For network input this seems to always work, but for file input we seeing odd behavior where some strings that are not JSON objects are being identified as if they were JSON objects.

Case in point is the msg "a,b,c,d,e,f”. If received over the network (imptcp) with a proper syslog header, this msg is correctly identified as not being a JSON object, and the value is applied to the $!smsg element in the if construct that follows.

However, if that same string is received in file input (imfile), the message is incorrectly identified as being a JSON object and the value is applied to the $!jmsg element. The same is true if the fields are pipe delimited as "a|b|c|d|e|f".

If I modify the msg to be wrapped in parenthesis, i.e. "(a,b,c,d,e,f)” or “(a|b|c|d|e|f)", then the string is identified properly in both cases.

Another thing that may be of significance is that if I add a syslog header to each line of the file input, and specify needparse=“on” in the file input definition, the issue goes away and the msg element is always identified properly.

This is v8.2102. I’m not sure if this is a bug or if I’m missing something fundamental about the differences between input types, so any advice or guidance is greatly appreciated.

Thanks all!






_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Odd parse_json Behavior With imfile Input [ In reply to ]
whenever you have a parsing problem, log the messages with the template
RSYSLOG_DebugFormat

90%+ of the time, we find that the contents of the variables are not what you
are expecting, and the rest of the time, seeing the debug output lets us figure
out what is going wrong with the parser

Also, what version of rsyslog are you running?

David Lang

On Thu, 17 Jun 2021, John Chivian via
rsyslog wrote:

> Date: Thu, 17 Jun 2021 11:05:21 -0500
> From: John Chivian via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: John Chivian <jchivian@chivian.com>
> Subject: [rsyslog] Odd parse_json Behavior With imfile Input
>
> I am using the parse_json function to determine whether or not the msg element is a JSON object.
>
> set $.isjson = parse_json($msg, "\$!parsed");
>
> if ($.isjson == 0) then {
> set $!jmsg = $msg;
> set $!smsg = "";
> } else {
> set $!jmsg = "null";
> set $!smsg = $msg;
> }
>
>
> For network input this seems to always work, but for file input we seeing odd behavior where some strings that are not JSON objects are being identified as if they were JSON objects.
>
> Case in point is the msg "a,b,c,d,e,f”. If received over the network (imptcp) with a proper syslog header, this msg is correctly identified as not being a JSON object, and the value is applied to the $!smsg element in the if construct that follows.
>
> However, if that same string is received in file input (imfile), the message is incorrectly identified as being a JSON object and the value is applied to the $!jmsg element. The same is true if the fields are pipe delimited as "a|b|c|d|e|f".
>
> If I modify the msg to be wrapped in parenthesis, i.e. "(a,b,c,d,e,f)” or “(a|b|c|d|e|f)", then the string is identified properly in both cases.
>
> Another thing that may be of significance is that if I add a syslog header to each line of the file input, and specify needparse=“on” in the file input definition, the issue goes away and the msg element is always identified properly.
>
> This is v8.2102. I’m not sure if this is a bug or if I’m missing something fundamental about the differences between input types, so any advice or guidance is greatly appreciated.
>
> Thanks all!
>
>
>
>
>
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Odd parse_json Behavior With imfile Input [ In reply to ]
Hi David:

I wish I could say that was the case, but the msg object is exactly what it should be. Here’s the output in error...

"string": "", "json": 206|domain1\user1|SEC_ACCOUNTING_CONSULTANT|N|2019-10-21 15:46:33.487|domain2\user2|10.224.129.254

…and here’s the output with parentheses added…

"string": "(206|domain1\\user1|SEC_ACCOUNTING_CONSULTANT|N|2019-10-21 15:46:33.487|domain2\\user2|10.224.129.254)", "json": "null"

…produced by…

constant(value="\"string\": \"") property(name="$!smsg" format="jsonr") constant(value="\",")
constant(value="\"json\": ") property(name="$!jmsg")
As you can see the only difference in msg is the added parenthesis, and both represent exactly what msg should be.


Regards,



> On Jun 17, 2021, at 11:16, David Lang <david@lang.hm> wrote:
>
> whenever you have a parsing problem, log the messages with the template RSYSLOG_DebugFormat
>
> 90%+ of the time, we find that the contents of the variables are not what you are expecting, and the rest of the time, seeing the debug output lets us figure out what is going wrong with the parser
>
> Also, what version of rsyslog are you running?
>
> David Lang
>
> On Thu, 17 Jun 2021, John Chivian via rsyslog wrote:
>
>> Date: Thu, 17 Jun 2021 11:05:21 -0500
>> From: John Chivian via rsyslog <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: John Chivian <jchivian@chivian.com>
>> Subject: [rsyslog] Odd parse_json Behavior With imfile Input
>> I am using the parse_json function to determine whether or not the msg element is a JSON object.
>>
>> set $.isjson = parse_json($msg, "\$!parsed");
>>
>> if ($.isjson == 0) then {
>> set $!jmsg = $msg;
>> set $!smsg = "";
>> } else {
>> set $!jmsg = "null";
>> set $!smsg = $msg;
>> }
>>
>>
>> For network input this seems to always work, but for file input we seeing odd behavior where some strings that are not JSON objects are being identified as if they were JSON objects.
>>
>> Case in point is the msg "a,b,c,d,e,f”. If received over the network (imptcp) with a proper syslog header, this msg is correctly identified as not being a JSON object, and the value is applied to the $!smsg element in the if construct that follows.
>>
>> However, if that same string is received in file input (imfile), the message is incorrectly identified as being a JSON object and the value is applied to the $!jmsg element. The same is true if the fields are pipe delimited as "a|b|c|d|e|f".
>> If I modify the msg to be wrapped in parenthesis, i.e. "(a,b,c,d,e,f)” or “(a|b|c|d|e|f)", then the string is identified properly in both cases.
>>
>> Another thing that may be of significance is that if I add a syslog header to each line of the file input, and specify needparse=“on” in the file input definition, the issue goes away and the msg element is always identified properly.
>>
>> This is v8.2102. I’m not sure if this is a bug or if I’m missing something fundamental about the differences between input types, so any advice or guidance is greatly appreciated.
>>
>> Thanks all!
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Odd parse_json Behavior With imfile Input [ In reply to ]
what is msg and rawmsg in these two cases?

what you are showing is the output, what we need to look at is the input to be
parsed.

David Lang

On Thu, 17 Jun 2021, John Chivian wrote:

> Date: Thu, 17 Jun 2021 11:37:14 -0500
> From: John Chivian <jchivian@chivian.com>
> To: David Lang <david@lang.hm>
> Cc: John Chivian via rsyslog <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] Odd parse_json Behavior With imfile Input
>
> Hi David:
>
> I wish I could say that was the case, but the msg object is exactly what it should be. Here’s the output in error...
>
> "string": "", "json": 206|domain1\user1|SEC_ACCOUNTING_CONSULTANT|N|2019-10-21 15:46:33.487|domain2\user2|10.224.129.254
>
> …and here’s the output with parentheses added…
>
> "string": "(206|domain1\\user1|SEC_ACCOUNTING_CONSULTANT|N|2019-10-21 15:46:33.487|domain2\\user2|10.224.129.254)", "json": "null"
>
> …produced by…
>
> constant(value="\"string\": \"") property(name="$!smsg" format="jsonr") constant(value="\",")
> constant(value="\"json\": ") property(name="$!jmsg")
> As you can see the only difference in msg is the added parenthesis, and both represent exactly what msg should be.
>
>
> Regards,
>
>
>
>> On Jun 17, 2021, at 11:16, David Lang <david@lang.hm> wrote:
>>
>> whenever you have a parsing problem, log the messages with the template RSYSLOG_DebugFormat
>>
>> 90%+ of the time, we find that the contents of the variables are not what you are expecting, and the rest of the time, seeing the debug output lets us figure out what is going wrong with the parser
>>
>> Also, what version of rsyslog are you running?
>>
>> David Lang
>>
>> On Thu, 17 Jun 2021, John Chivian via rsyslog wrote:
>>
>>> Date: Thu, 17 Jun 2021 11:05:21 -0500
>>> From: John Chivian via rsyslog <rsyslog@lists.adiscon.com>
>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>> Cc: John Chivian <jchivian@chivian.com>
>>> Subject: [rsyslog] Odd parse_json Behavior With imfile Input
>>> I am using the parse_json function to determine whether or not the msg element is a JSON object.
>>>
>>> set $.isjson = parse_json($msg, "\$!parsed");
>>>
>>> if ($.isjson == 0) then {
>>> set $!jmsg = $msg;
>>> set $!smsg = "";
>>> } else {
>>> set $!jmsg = "null";
>>> set $!smsg = $msg;
>>> }
>>>
>>>
>>> For network input this seems to always work, but for file input we seeing odd behavior where some strings that are not JSON objects are being identified as if they were JSON objects.
>>>
>>> Case in point is the msg "a,b,c,d,e,f”. If received over the network (imptcp) with a proper syslog header, this msg is correctly identified as not being a JSON object, and the value is applied to the $!smsg element in the if construct that follows.
>>>
>>> However, if that same string is received in file input (imfile), the message is incorrectly identified as being a JSON object and the value is applied to the $!jmsg element. The same is true if the fields are pipe delimited as "a|b|c|d|e|f".
>>> If I modify the msg to be wrapped in parenthesis, i.e. "(a,b,c,d,e,f)” or “(a|b|c|d|e|f)", then the string is identified properly in both cases.
>>>
>>> Another thing that may be of significance is that if I add a syslog header to each line of the file input, and specify needparse=“on” in the file input definition, the issue goes away and the msg element is always identified properly.
>>>
>>> This is v8.2102. I’m not sure if this is a bug or if I’m missing something fundamental about the differences between input types, so any advice or guidance is greatly appreciated.
>>>
>>> Thanks all!
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.