Mailing List Archive: customizing and aggregating impstats

customizing and aggregating impstats

Feb 1, 2019, 12:37 AM

Post #1 of 8 (544 views)

Hi,

As we know /impstats/ module is used to fetch various stats of rsyslog
for all the objects like actions, queues, etc.

For each object separate stats log line is generated like,

/Mon Jan 21 06:16:09 2019: @cee: { "*name*": "action-1-omelasticsearch",
"_origin_": "core.action", "_processed_": 0, "_failed_": 0,
"_suspended_": 0, "_suspended.duration_": 0, "_resumed_": 0 }/

/Mon Jan 21 06:16:09 2019: @cee: { "*name*": "action-1-omelasticsearch
queue", "_origi__n_": "core.queue", "_size_": 0, "_enqueued_": 0,
"_full_": 0, "_discarded.full_": 0, "_discarded.nf_": 0, "_maxqsize_": 0 }/

I have couple queries on this,

1) Is there a way to make the stats fields/key unique like, can the
'processed' field be perpended with value of name field ?

e.g. from '/processed/' to '/action-1-omelasticsearch.processed'/

2) For aggregate stats processing, looking for a way to sum up the stats
of all actions and queues for particular ruleset so that it eases in
further analysis.

for this I tried exploring writing python code, but rsyslog has external
module support for the same and which is out of the proc execution.

What I am looking is more like in process thing simillar to syslog-ng.

Any help on the above queries will be greatly appreciated as I am new to
rsyslog and may have missed some setting which can be easily used for
this kind of requirement.

Thanks,

Vishal.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: customizing and aggregating impstats [ In reply to ]

rgerhards at hq

Feb 1, 2019, 12:44 AM

Post #2 of 8 (544 views)

Permalink

El vie., 1 feb. 2019 a las 9:37, vishal via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> Hi,
>
> As we know /impstats/ module is used to fetch various stats of rsyslog
> for all the objects like actions, queues, etc.
>
> For each object separate stats log line is generated like,
>
> /Mon Jan 21 06:16:09 2019: @cee: { "*name*": "action-1-omelasticsearch",
> "_origin_": "core.action", "_processed_": 0, "_failed_": 0,
> "_suspended_": 0, "_suspended.duration_": 0, "_resumed_": 0 }/
>
> /Mon Jan 21 06:16:09 2019: @cee: { "*name*": "action-1-omelasticsearch
> queue", "_origi__n_": "core.queue", "_size_": 0, "_enqueued_": 0,
> "_full_": 0, "_discarded.full_": 0, "_discarded.nf_": 0, "_maxqsize_": 0 }/
>
> I have couple queries on this,
>
> 1) Is there a way to make the stats fields/key unique like, can the
> 'processed' field be perpended with value of name field ?
>
> e.g. from '/processed/' to '/action-1-omelasticsearch.processed'/
>

That would be possible but very verbose and redundant. You would need
to prefix all of the fields with the name, because they all belong to
the same name.

>
> 2) For aggregate stats processing, looking for a way to sum up the stats
> of all actions and queues for particular ruleset so that it eases in
> further analysis.
>
> for this I tried exploring writing python code, but rsyslog has external
> module support for the same and which is out of the proc execution.
>
> What I am looking is more like in process thing simillar to syslog-ng.

Can you point to how exactly you would like to have it?

Rainer
>
>
> Any help on the above queries will be greatly appreciated as I am new to
> rsyslog and may have missed some setting which can be easily used for
> this kind of requirement.
>
> Thanks,
>
> Vishal.
>
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: customizing and aggregating impstats [ In reply to ]

rsyslog at lists

Feb 1, 2019, 12:54 AM

Post #3 of 8 (544 views)

Permalink

On 01/02/19 2:14 PM, Rainer Gerhards wrote:
> El vie., 1 feb. 2019 a las 9:37, vishal via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:

Thanks Rainer for the quick reply.

Please see inline.

>> Hi,
>>
>> As we know /impstats/ module is used to fetch various stats of rsyslog
>> for all the objects like actions, queues, etc.
>>
>> For each object separate stats log line is generated like,
>>
>> /Mon Jan 21 06:16:09 2019: @cee: { "*name*": "action-1-omelasticsearch",
>> "_origin_": "core.action", "_processed_": 0, "_failed_": 0,
>> "_suspended_": 0, "_suspended.duration_": 0, "_resumed_": 0 }/
>>
>> /Mon Jan 21 06:16:09 2019: @cee: { "*name*": "action-1-omelasticsearch
>> queue", "_origi__n_": "core.queue", "_size_": 0, "_enqueued_": 0,
>> "_full_": 0, "_discarded.full_": 0, "_discarded.nf_": 0, "_maxqsize_": 0 }/
>>
>> I have couple queries on this,
>>
>> 1) Is there a way to make the stats fields/key unique like, can the
>> 'processed' field be perpended with value of name field ?
>>
>> e.g. from '/processed/' to '/action-1-omelasticsearch.processed'/
>>
> That would be possible but very verbose and redundant. You would need
> to prefix all of the fields with the name, because they all belong to
> the same name.

how to do it ? through some kind of template, could you please provide
some example?

>
>> 2) For aggregate stats processing, looking for a way to sum up the stats
>> of all actions and queues for particular ruleset so that it eases in
>> further analysis.
>>
>> for this I tried exploring writing python code, but rsyslog has external
>> module support for the same and which is out of the proc execution.
>>
>> What I am looking is more like in process thing simillar to syslog-ng.
> Can you point to how exactly you would like to have it?
>
> Rainer

is it possible to define a python/anyOtherLanguage code block inside
conf file and should be able to call the functions through some ruleset
as a part of action(s). so that whatever changes made to messages are
serialized and aggregated/modified messages are sent further to output
module.

>>
>> Any help on the above queries will be greatly appreciated as I am new to
>> rsyslog and may have missed some setting which can be easily used for
>> this kind of requirement.
>>
>> Thanks,
>>
>> Vishal.
>>
>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: customizing and aggregating impstats [ In reply to ]

rgerhards at hq

Feb 1, 2019, 1:34 AM

Post #4 of 8 (543 views)

Permalink

> >> As we know /impstats/ module is used to fetch various stats of rsyslog
> >> for all the objects like actions, queues, etc.
> >>
> >> For each object separate stats log line is generated like,
> >>
> >> /Mon Jan 21 06:16:09 2019: @cee: { "*name*": "action-1-omelasticsearch",
> >> "_origin_": "core.action", "_processed_": 0, "_failed_": 0,
> >> "_suspended_": 0, "_suspended.duration_": 0, "_resumed_": 0 }/
> >>
> >> /Mon Jan 21 06:16:09 2019: @cee: { "*name*": "action-1-omelasticsearch
> >> queue", "_origi__n_": "core.queue", "_size_": 0, "_enqueued_": 0,
> >> "_full_": 0, "_discarded.full_": 0, "_discarded.nf_": 0, "_maxqsize_": 0 }/
> >>
> >> I have couple queries on this,
> >>
> >> 1) Is there a way to make the stats fields/key unique like, can the
> >> 'processed' field be perpended with value of name field ?
> >>
> >> e.g. from '/processed/' to '/action-1-omelasticsearch.processed'/
> >>
> > That would be possible but very verbose and redundant. You would need
> > to prefix all of the fields with the name, because they all belong to
> > the same name.
>
> how to do it ? through some kind of template, could you please provide
> some example?

No, I mean the code would need to do that. We could add an option, but
I question that it really is the right thing to do.

> >> 2) For aggregate stats processing, looking for a way to sum up the stats
> >> of all actions and queues for particular ruleset so that it eases in
> >> further analysis.
> >>
> >> for this I tried exploring writing python code, but rsyslog has external
> >> module support for the same and which is out of the proc execution.
> >>
> >> What I am looking is more like in process thing simillar to syslog-ng.
> > Can you point to how exactly you would like to have it?
> >
> > Rainer
>
> is it possible to define a python/anyOtherLanguage code block inside
> conf file and should be able to call the functions through some ruleset
> as a part of action(s). so that whatever changes made to messages are
> serialized and aggregated/modified messages are sent further to output
> module.

I see, we have a more generic and secure approach. See

https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmexternal.html

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: customizing and aggregating impstats [ In reply to ]

rsyslog at lists

Feb 1, 2019, 2:26 AM

Post #5 of 8 (542 views)

Permalink

take a look at the dyn_stats() capability, it lets you create your own stats
that get reported as part of the pstats output.

https://www.rsyslog.com/doc/master/configuration/dyn_stats.html

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: customizing and aggregating impstats [ In reply to ]

jchivian at chivian

Feb 1, 2019, 3:18 AM

Post #6 of 8 (542 views)

Permalink

I have been using mmexternal to modify pstats messages for quite some time.

syslog.* {
if ($programname == "rsyslog-pstats") then {
    action (
      type="mmexternal"
      name="normalize-pstats"
      binary="/path/to/binary"
      interface.input="msg"
    )
    call to_pstats_siem
    call to_pstats_debug
} else {
    action (
      type="omfile"
      name="syslog"
      file="/var/log/syslog"
    )
}
}

On 2/1/19 4:26 AM, David Lang via rsyslog wrote:
> take a look at the dyn_stats() capability, it lets you create your own
> stats that get reported as part of the pstats output.
>
> https://www.rsyslog.com/doc/master/configuration/dyn_stats.html
>
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: customizing and aggregating impstats [ In reply to ]

rsyslog at lists

Feb 3, 2019, 10:53 PM

Post #7 of 8 (534 views)

Permalink

Thanks John, will explore mmexternal further.

Just a quick query, can does mmexternal works perfectly fine in pipeline?

like,

imudp -> mmexternal -> omelasticsearch

Thanks,

Vishal Patil.

On 01/02/19 4:48 PM, John Chivian wrote:
> I have been using mmexternal to modify pstats messages for quite some
> time.
>
> syslog.* {
> if ($programname == "rsyslog-pstats") then {
>     action (
>       type="mmexternal"
>       name="normalize-pstats"
>       binary="/path/to/binary"
>       interface.input="msg"
>     )
>     call to_pstats_siem
>     call to_pstats_debug
> } else {
>     action (
>       type="omfile"
>       name="syslog"
>       file="/var/log/syslog"
>     )
> }
> }
>
>
>
> On 2/1/19 4:26 AM, David Lang via rsyslog wrote:
>> take a look at the dyn_stats() capability, it lets you create your
>> own stats that get reported as part of the pstats output.
>>
>> https://www.rsyslog.com/doc/master/configuration/dyn_stats.html
>>
>> David Lang
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST if you DON'T LIKE THAT.
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: customizing and aggregating impstats [ In reply to ]

rgerhards at hq

Feb 4, 2019, 2:29 AM

Post #8 of 8 (533 views)

Permalink

El lun., 4 feb. 2019 a las 7:53, vishal via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> Thanks John, will explore mmexternal further.
>
> Just a quick query, can does mmexternal works perfectly fine in pipeline?
>
> like,
>
> imudp -> mmexternal -> omelasticsearch

yes, that it was written for.

HTH
Rainer
>
>
> Thanks,
>
> Vishal Patil.
>
> On 01/02/19 4:48 PM, John Chivian wrote:
> > I have been using mmexternal to modify pstats messages for quite some
> > time.
> >
> > syslog.* {
> > if ($programname == "rsyslog-pstats") then {
> > action (
> > type="mmexternal"
> > name="normalize-pstats"
> > binary="/path/to/binary"
> > interface.input="msg"
> > )
> > call to_pstats_siem
> > call to_pstats_debug
> > } else {
> > action (
> > type="omfile"
> > name="syslog"
> > file="/var/log/syslog"
> > )
> > }
> > }
> >
> >
> >
> > On 2/1/19 4:26 AM, David Lang via rsyslog wrote:
> >> take a look at the dyn_stats() capability, it lets you create your
> >> own stats that get reported as part of the pstats output.
> >>
> >> https://www.rsyslog.com/doc/master/configuration/dyn_stats.html
> >>
> >> David Lang
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> >> POST if you DON'T LIKE THAT.
> >
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> > if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.