Mailing List Archive: impstats json nested fields

impstats json nested fields

Jan 3, 2022, 2:42 AM

Post #1 of 3 (344 views)

Hi
Wish happy and productive new year to everyone.

I am trying to send impstats to Elasticsearch however due to particular elasticsearch cluster configuration I cannot use rsyslog elasticsearch module. I am forced to use filebeat.

Essestially what i am doing is to output impstats on a file (in json format) and then use filebeat to forward the stats. I noticed the json stats generated from impstats are not structured. For example, the "discarded" filed in the following line is not structured as an ES object.

{ "name": "monitoring[DA]", "origin": "core.queue", "size": 0, "enqueued": 0, "full": 0, "discarded.full": 0, "discarded.nf": 0, "maxqsize": 0 }

Is there a way impstat could structure nested fields like:

"discarded.full": 0, "discarded.nf": 0

to

"discarded": {"full": 0, "nf": 0}

Thanks
D.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: impstats json nested fields [ In reply to ]

rsyslog at lists

Jan 3, 2022, 4:28 AM

Post #2 of 3 (344 views)

Permalink

discarded.full
discarded.nf
suspended.duration
ratelimit.discarded
ratelimit.numratelimiters
sessions.opened
sessions.closed
sessions.openfailed
bytes.received
bytes.decompressed
bytes.sent

…all fall into that category. When we had Elastic we transformed the dot "." in the cases above to an underscore "_" during transmission to solve that exact issue. Now that we have a “real” SIEM we simply tell it to extract .\"bytes.sent\"

I don’t think there’s really much else you can do.

Regards,

> On Jan 3, 2022, at 04:42, Dimi Onobodies via rsyslog <rsyslog@lists.adiscon.com> wrote:
>
> Hi
> Wish happy and productive new year to everyone.
>
> I am trying to send impstats to Elasticsearch however due to particular elasticsearch cluster configuration I cannot use rsyslog elasticsearch module. I am forced to use filebeat.
>
> Essestially what i am doing is to output impstats on a file (in json format) and then use filebeat to forward the stats. I noticed the json stats generated from impstats are not structured. For example, the "discarded" filed in the following line is not structured as an ES object.
>
> { "name": "monitoring[DA]", "origin": "core.queue", "size": 0, "enqueued": 0, "full": 0, "discarded.full": 0, "discarded.nf": 0, "maxqsize": 0 }
>
> Is there a way impstat could structure nested fields like:
>
> "discarded.full": 0, "discarded.nf": 0
>
> to
>
> "discarded": {"full": 0, "nf": 0}
>
>
> Thanks
> D.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: impstats json nested fields [ In reply to ]

rsyslog at lists

Jan 3, 2022, 10:43 AM

Post #3 of 3 (339 views)

Permalink

we've needed a message modification module that could fix this sort of thing

unflatten(".",$!) or similar for this case

this is also needed when you parse things with mmnormalize and want to extract
different values and have them in a nested json structure

rsyslog has no way to deal with a name that contains an embedded '!' for
example, but unflatten('!',$!) would solve that

I've misplaced my routd toit and haven't gotten it done, contributions welcome

David Lang

On Mon, 3 Jan 2022, John Chivian via rsyslog wrote:

> Date: Mon, 3 Jan 2022 06:28:16 -0600
> From: John Chivian via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: John Chivian <jchivian@chivian.com>
> Subject: Re: [rsyslog] impstats json nested fields
>
> discarded.full
> discarded.nf
> suspended.duration
> ratelimit.discarded
> ratelimit.numratelimiters
> sessions.opened
> sessions.closed
> sessions.openfailed
> bytes.received
> bytes.decompressed
> bytes.sent
>
> …all fall into that category. When we had Elastic we transformed the dot "." in the cases above to an underscore "_" during transmission to solve that exact issue. Now that we have a “real” SIEM we simply tell it to extract .\"bytes.sent\"
>
> I don’t think there’s really much else you can do.
>
> Regards,
>
>
>> On Jan 3, 2022, at 04:42, Dimi Onobodies via rsyslog <rsyslog@lists.adiscon.com> wrote:
>>
>> Hi
>> Wish happy and productive new year to everyone.
>>
>> I am trying to send impstats to Elasticsearch however due to particular elasticsearch cluster configuration I cannot use rsyslog elasticsearch module. I am forced to use filebeat.
>>
>> Essestially what i am doing is to output impstats on a file (in json format) and then use filebeat to forward the stats. I noticed the json stats generated from impstats are not structured. For example, the "discarded" filed in the following line is not structured as an ES object.
>>
>> { "name": "monitoring[DA]", "origin": "core.queue", "size": 0, "enqueued": 0, "full": 0, "discarded.full": 0, "discarded.nf": 0, "maxqsize": 0 }
>>
>> Is there a way impstat could structure nested fields like:
>>
>> "discarded.full": 0, "discarded.nf": 0
>>
>> to
>>
>> "discarded": {"full": 0, "nf": 0}
>>
>>
>> Thanks
>> D.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.