Mailing List Archive

JSON extraction
I am logging JSON that has a JSON field called msg that is a string that contains JSON. I would like to configure rsyslog to convert the JSON in the msg field to be JSON at the same level as the original message.


Example:


{"time":"2020-02-02T18:00:00", "mode":"imuxsock", "host":"venus", "msg":" {"name":"bill", "address":"123 Anywhere", "City":"Cleveland", "State":"Ohio", "zip":"12345"}"}


What I want logged is:


{"time":"2020-02-02T18:00:00", "mode":"imuxsock", "host":"venus", "name":"bill", "address":"123 Anywhere", "City":"Cleveland", "State":"Ohio", "zip":"12345"}


I've tried mmjsonparse but can't seem to get anywhere with it.



_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: JSON extraction [ In reply to ]
mmjsonparse is a pretty specialized use case, I would suggest that you look
at mmnormalize, particularly the ability to parse variables.

note that the quotes would need escaping with your example, what you posted
isn't actually valid JSON.

David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: JSON extraction [ In reply to ]
David is correct. If what you posted is actually your input…

{"time":"2020-02-02T18:00:00", "mode":"imuxsock", "host":"venus", "msg":" {"name":"bill", "address":"123 Anywhere", "City":"Cleveland", "State":"Ohio", "zip":"12345"}”}

…then you’re not likely to have much luck doing anything but a manual parse. The input should like like this…

{"time":"2020-02-02T18:00:00", "mode":"imuxsock", "host":"venus", "msg": {"name":"bill", "address":"123 Anywhere", "City":"Cleveland", "State":"Ohio", "zip":"12345”}}

…so that at least the structure is valid. If the intent is truly for the msg object to be a string, not a JSON object, then you must escape the quotes within it.

I would then argue that flattening the JSON object, such that the fields within are part of the top level, is a loss definition not an improvement.

Regards,


> On Feb 4, 2021, at 17:14, David Lang via rsyslog <rsyslog@lists.adiscon.com> wrote:
>
> mmjsonparse is a pretty specialized use case, I would suggest that you look at mmnormalize, particularly the ability to parse variables.
>
> note that the quotes would need escaping with your example, what you posted isn't actually valid JSON.
>
> David Lang
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: JSON extraction [ In reply to ]
escaped it would be

> {"time":"2020-02-02T18:00:00", "mode":"imuxsock", "host":"venus", "msg":" {\"name\":\"bill\", \"address\":\"123 Anywhere\", \"City\":\"Cleveland\", \"State\":\"Ohio\", \"zip\":\"12345\"}”}

David Lang

On Thu, 4 Feb 2021, John Chivian wrote:

> Date: Thu, 4 Feb 2021 17:49:07 -0600
> From: John Chivian <jchivian@chivian.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: David Lang <david@lang.hm>
> Subject: Re: [rsyslog] JSON extraction
>
> David is correct. If what you posted is actually your input…
>
> {"time":"2020-02-02T18:00:00", "mode":"imuxsock", "host":"venus", "msg":" {"name":"bill", "address":"123 Anywhere", "City":"Cleveland", "State":"Ohio", "zip":"12345"}”}
>
> …then you’re not likely to have much luck doing anything but a manual parse. The input should like like this…
>
> {"time":"2020-02-02T18:00:00", "mode":"imuxsock", "host":"venus", "msg": {"name":"bill", "address":"123 Anywhere", "City":"Cleveland", "State":"Ohio", "zip":"12345”}}
>
> …so that at least the structure is valid. If the intent is truly for the msg object to be a string, not a JSON object, then you must escape the quotes within it.
>
> I would then argue that flattening the JSON object, such that the fields within are part of the top level, is a loss definition not an improvement.
>
> Regards,
>
>
>> On Feb 4, 2021, at 17:14, David Lang via rsyslog <rsyslog@lists.adiscon.com> wrote:
>>
>> mmjsonparse is a pretty specialized use case, I would suggest that you look at mmnormalize, particularly the ability to parse variables.
>>
>> note that the quotes would need escaping with your example, what you posted isn't actually valid JSON.
>>
>> David Lang
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: JSON extraction [ In reply to ]
I haven't tried this, but as an idea.

Use mmnormalize/json twice:
1. parse the "upper" level, giving you msg
2. parse again, this time using msg as input

Just an idea...
Rainer

El vie, 5 feb 2021 a las 0:49, John Chivian via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> David is correct. If what you posted is actually your input…
>
> {"time":"2020-02-02T18:00:00", "mode":"imuxsock", "host":"venus", "msg":" {"name":"bill", "address":"123 Anywhere", "City":"Cleveland", "State":"Ohio", "zip":"12345"}”}
>
> …then you’re not likely to have much luck doing anything but a manual parse. The input should like like this…
>
> {"time":"2020-02-02T18:00:00", "mode":"imuxsock", "host":"venus", "msg": {"name":"bill", "address":"123 Anywhere", "City":"Cleveland", "State":"Ohio", "zip":"12345”}}
>
> …so that at least the structure is valid. If the intent is truly for the msg object to be a string, not a JSON object, then you must escape the quotes within it.
>
> I would then argue that flattening the JSON object, such that the fields within are part of the top level, is a loss definition not an improvement.
>
> Regards,
>
>
> > On Feb 4, 2021, at 17:14, David Lang via rsyslog <rsyslog@lists.adiscon.com> wrote:
> >
> > mmjsonparse is a pretty specialized use case, I would suggest that you look at mmnormalize, particularly the ability to parse variables.
> >
> > note that the quotes would need escaping with your example, what you posted isn't actually valid JSON.
> >
> > David Lang
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: JSON extraction [ In reply to ]
I’m working on this again - I’m having trouble finding a good example of mmnormalize - can someone point me to a good example?

Thanks

> On Feb 5, 2021, at 5:36 AM, Rainer Gerhards via rsyslog <rsyslog@lists.adiscon.com> wrote:
>
> I haven't tried this, but as an idea.
>
> Use mmnormalize/json twice:
> 1. parse the "upper" level, giving you msg
> 2. parse again, this time using msg as input
>
> Just an idea...
> Rainer
>
> El vie, 5 feb 2021 a las 0:49, John Chivian via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
>>
>> David is correct. If what you posted is actually your input…
>>
>> {"time":"2020-02-02T18:00:00", "mode":"imuxsock", "host":"venus", "msg":" {"name":"bill", "address":"123 Anywhere", "City":"Cleveland", "State":"Ohio", "zip":"12345"}”}
>>
>> …then you’re not likely to have much luck doing anything but a manual parse. The input should like like this…
>>
>> {"time":"2020-02-02T18:00:00", "mode":"imuxsock", "host":"venus", "msg": {"name":"bill", "address":"123 Anywhere", "City":"Cleveland", "State":"Ohio", "zip":"12345”}}
>>
>> …so that at least the structure is valid. If the intent is truly for the msg object to be a string, not a JSON object, then you must escape the quotes within it.
>>
>> I would then argue that flattening the JSON object, such that the fields within are part of the top level, is a loss definition not an improvement.
>>
>> Regards,
>>
>>
>>> On Feb 4, 2021, at 17:14, David Lang via rsyslog <rsyslog@lists.adiscon.com> wrote:
>>>
>>> mmjsonparse is a pretty specialized use case, I would suggest that you look at mmnormalize, particularly the ability to parse variables.
>>>
>>> note that the quotes would need escaping with your example, what you posted isn't actually valid JSON.
>>>
>>> David Lang
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: JSON extraction [ In reply to ]
can you provide a sample log message? then it's easy to show a sample rule that
matches it.

David Lang

On Tue, 22 Jun 2021, ListKP via rsyslog wrote:

> Date: Tue, 22 Jun 2021 11:06:39 -0400
> From: ListKP via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: ListKP <listkp@icloud.com>
> Subject: Re: [rsyslog] JSON extraction
>
> I’m working on this again - I’m having trouble finding a good example of mmnormalize - can someone point me to a good example?
>
> Thanks
>
>> On Feb 5, 2021, at 5:36 AM, Rainer Gerhards via rsyslog <rsyslog@lists.adiscon.com> wrote:
>>
>> I haven't tried this, but as an idea.
>>
>> Use mmnormalize/json twice:
>> 1. parse the "upper" level, giving you msg
>> 2. parse again, this time using msg as input
>>
>> Just an idea...
>> Rainer
>>
>> El vie, 5 feb 2021 a las 0:49, John Chivian via rsyslog
>> (<rsyslog@lists.adiscon.com>) escribió:
>>>
>>> David is correct. If what you posted is actually your input…
>>>
>>> {"time":"2020-02-02T18:00:00", "mode":"imuxsock", "host":"venus", "msg":" {"name":"bill", "address":"123 Anywhere", "City":"Cleveland", "State":"Ohio", "zip":"12345"}”}
>>>
>>> …then you’re not likely to have much luck doing anything but a manual parse. The input should like like this…
>>>
>>> {"time":"2020-02-02T18:00:00", "mode":"imuxsock", "host":"venus", "msg": {"name":"bill", "address":"123 Anywhere", "City":"Cleveland", "State":"Ohio", "zip":"12345”}}
>>>
>>> …so that at least the structure is valid. If the intent is truly for the msg object to be a string, not a JSON object, then you must escape the quotes within it.
>>>
>>> I would then argue that flattening the JSON object, such that the fields within are part of the top level, is a loss definition not an improvement.
>>>
>>> Regards,
>>>
>>>
>>>> On Feb 4, 2021, at 17:14, David Lang via rsyslog <rsyslog@lists.adiscon.com> wrote:
>>>>
>>>> mmjsonparse is a pretty specialized use case, I would suggest that you look at mmnormalize, particularly the ability to parse variables.
>>>>
>>>> note that the quotes would need escaping with your example, what you posted isn't actually valid JSON.
>>>>
>>>> David Lang
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.