Mailing List Archive

Information about incoming logs.
Hello rsyslog community,

I'm new to rsyslog, but have already successfully configured some machines to send their logs using mutual TLS authentication.
So far so good, but I have some questions :

How can I see from the rsyslog server side how many and which machines send logs ?
With the command "systemctl status rsyslog" I have a view on what happens and which machines have problems, but it is not very clear.
Can I somehow see which machines are successfully communicating and sending their logs ?
Another very interesting information for me, would be to be able to see the time of the last update for each machine.

For example:
Domain Status Last update
domain1.com [Connection OK] 04:04:33
domain2.com [Connection FAILED] 02:32:03
domain3.com [Connection OK] 04:02:12
....

Can I get this kind of information from the rsyslog server ?

Thanks in advance.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Information about incoming logs. [ In reply to ]
take a look at the impstats module, and the ability for the imtcp module to
record per-sender stats

if that isn't enough, you can create your own stats via the dyn_stats()
functions

David Lang

On Tue, 16 Feb 2021, odrzen via rsyslog wrote:

> Date: Tue, 16 Feb 2021 23:04:40 +0000
> From: odrzen via rsyslog <rsyslog@lists.adiscon.com>
> Reply-To: odrzen <odrzen@protonmail.com>,
> rsyslog-users <rsyslog@lists.adiscon.com>
> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
> Cc: odrzen <odrzen@protonmail.com>
> Subject: [rsyslog] Information about incoming logs.
>
> Hello rsyslog community,
>
> I'm new to rsyslog, but have already successfully configured some machines to send their logs using mutual TLS authentication.
> So far so good, but I have some questions :
>
> How can I see from the rsyslog server side how many and which machines send logs ?
> With the command "systemctl status rsyslog" I have a view on what happens and which machines have problems, but it is not very clear.
> Can I somehow see which machines are successfully communicating and sending their logs ?
> Another very interesting information for me, would be to be able to see the time of the last update for each machine.
>
> For example:
> Domain Status Last update
> domain1.com [Connection OK] 04:04:33
> domain2.com [Connection FAILED] 02:32:03
> domain3.com [Connection OK] 04:02:12
> ....
>
> Can I get this kind of information from the rsyslog server ?
>
> Thanks in advance.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Information about incoming logs. [ In reply to ]
You can also simply check TCP connections' state. One caveat is that in
case of rarely-talking sources they might not initiate connection until
they have data to send.

Mariusz Kruk
Ekspert ds. Bezpiecze?stwa IT
COMP S.A.
Pion Cyberbezpiecze?stwa i Zarz?dzania Ryzykiem
e-mail: mariusz.kruk@comp.com.pl
e-mail: mariusz.kruk@safecomp.com
tel: +48 608 623 299

On 17.02.2021 00:29, David Lang via rsyslog wrote:
> take a look at the impstats module, and the ability for the imtcp
> module to record per-sender stats
>
> if that isn't enough, you can create your own stats via the
> dyn_stats() functions
>
> David Lang
>
> On Tue, 16 Feb 2021, odrzen via rsyslog wrote:
>
>> Date: Tue, 16 Feb 2021 23:04:40 +0000
>> From: odrzen via rsyslog <rsyslog@lists.adiscon.com>
>> Reply-To: odrzen <odrzen@protonmail.com>,
>>     rsyslog-users <rsyslog@lists.adiscon.com>
>> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
>> Cc: odrzen <odrzen@protonmail.com>
>> Subject: [rsyslog] Information about incoming logs.
>>
>> Hello rsyslog community,
>>
>> I'm new to rsyslog, but have already successfully configured some
>> machines to send their logs using mutual TLS authentication.
>> So far so good, but I have some questions :
>>
>> How can I see from the rsyslog server side how many and which
>> machines send logs ?
>> With the command "systemctl status rsyslog" I have a view on what
>> happens and which machines have problems, but it is not very clear.
>> Can I somehow see which machines are successfully communicating and
>> sending their logs ?
>> Another very interesting information for me, would be to be able to
>> see the time of the last update for each machine.
>>
>> For example:
>> Domain Status Last update
>> domain1.com [Connection OK] 04:04:33
>> domain2.com [Connection FAILED] 02:32:03
>> domain3.com [Connection OK] 04:02:12
>> ....
>>
>> Can I get this kind of information from the rsyslog server ?
>>
>> Thanks in advance.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST if you DON'T LIKE THAT.
>>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Information about incoming logs. [ In reply to ]
So far, looking for what you are telling us, I have been able to add the following configuration `/etc/rsyslog.conf` :
```
module(load="impstats"
interval="600"
severity="7"
log.syslog="off"
log.file="/var/log/rsyslog-stats/stats.log")

global(senders.keepTrack="on")
dyn_stats(name="msg_per_host")
```

and here are the results I get every 10 minutes:
```
Fri Feb 19 00:52:08 2021: global: origin=dynstats msg_per_host.ops_overflow=0 msg_per_host.new_metric_add=0 msg_per_host.no_metric=0 msg_per_host.metrics_purged=0 msg_per_host.ops_ignored=0 msg_per_host.purge_triggered=0
Fri Feb 19 00:52:08 2021: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
Fri Feb 19 00:52:08 2021: dynafile cache Auditlog: origin=omfile requests=1638 level0=1020 missed=76 evicted=66 maxused=10 closetimeouts=0
Fri Feb 19 00:52:08 2021: action 0: origin=core.action processed=1638 failed=0 suspended=0 suspended.duration=0 resumed=0
Fri Feb 19 00:52:08 2021: dynafile cache RemoteLogs: origin=omfile requests=75872 level0=8240 missed=941 evicted=931 maxused=10 closetimeouts=0
Fri Feb 19 00:52:08 2021: action 1: origin=core.action processed=75872 failed=0 suspended=0 suspended.duration=0 resumed=0
Fri Feb 19 00:52:08 2021: action 2: origin=core.action processed=74218 failed=0 suspended=0 suspended.duration=0 resumed=0
Fri Feb 19 00:52:08 2021: action 3: origin=core.action processed=1638 failed=0 suspended=0 suspended.duration=0 resumed=0
Fri Feb 19 00:52:08 2021: action 4: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Fri Feb 19 00:52:08 2021: action 5: origin=core.action processed=16 failed=0 suspended=0 suspended.duration=0 resumed=0
Fri Feb 19 00:52:08 2021: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Fri Feb 19 00:52:08 2021: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Fri Feb 19 00:52:08 2021: action 8: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Fri Feb 19 00:52:08 2021: msg_per_host: origin=dynstats.bucket
Fri Feb 19 00:52:08 2021: imudp(*:514): origin=imudp submitted=0
Fri Feb 19 00:52:08 2021: imudp(*:514): origin=imudp submitted=0
Fri Feb 19 00:52:08 2021: imtcp(6514): origin=imtcp submitted=75441
Fri Feb 19 00:52:08 2021: resource-usage: origin=impstats utime=6334580 stime=7795002 maxrss=9256 minflt=2781 majflt=6 inblock=1320 oublock=98680 nvcsw=146787 nivcsw=40
Fri Feb 19 00:52:08 2021: main Q: origin=core.queue size=0 enqueued=75872 full=0 discarded.full=0 discarded.nf=0 maxqsize=85
Fri Feb 19 00:52:08 2021: imudp(w0): origin=imudp called.recvmmsg=0 called.recvmsg=0 msgs.received=0
Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_1.com messages=21
Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_2.com messages=21814
Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_3.com messages=5096
Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_4.com messages=25
Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_5.com messages=37
Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_6.com messages=151
Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_7.com messages=13
Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_8.com messages=7460
Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_9.com messages=21
Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_10.com messages=5118
Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_12.com messages=5099
Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_13.com messages=14
Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_14.com messages=7820
Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_15.com messages=123
Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_16.com messages=15170
Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_17.com messages=7459
```

About the "the imtcp module to record per-sender stats", I'm not sure I found what you mean, because I only found this parameter in this `dyn_stats(name="msg_per_host")` order.

About dyn_stats(), how can I create my "own" statistics ? For example, can I create reports per/hour or per/days ? In addition, can I include more information in these reports ?

In addition, can I see if there are problems with some machines with connection ( for example wrong certificates ) or bottleneck problems ?


Thank you very much for your time. I appreciate your help.


??????? Original Message ???????
On Wednesday, February 17, 2021 1:29 AM, David Lang <david@lang.hm> wrote:

> take a look at the impstats module, and the ability for the imtcp module to
> record per-sender stats
>
> if that isn't enough, you can create your own stats via the dyn_stats()
> functions
>
> David Lang
>
> On Tue, 16 Feb 2021, odrzen via rsyslog wrote:
>
> > Date: Tue, 16 Feb 2021 23:04:40 +0000
> > From: odrzen via rsyslog rsyslog@lists.adiscon.com
> > Reply-To: odrzen odrzen@protonmail.com,
> > rsyslog-users rsyslog@lists.adiscon.com
> > To: "rsyslog@lists.adiscon.com" rsyslog@lists.adiscon.com
> > Cc: odrzen odrzen@protonmail.com
> > Subject: [rsyslog] Information about incoming logs.
> > Hello rsyslog community,
> > I'm new to rsyslog, but have already successfully configured some machines to send their logs using mutual TLS authentication.
> > So far so good, but I have some questions :
> > How can I see from the rsyslog server side how many and which machines send logs ?
> > With the command "systemctl status rsyslog" I have a view on what happens and which machines have problems, but it is not very clear.
> > Can I somehow see which machines are successfully communicating and sending their logs ?
> > Another very interesting information for me, would be to be able to see the time of the last update for each machine.
> > For example:
> > Domain Status Last update
> > domain1.com [Connection OK] 04:04:33
> > domain2.com [Connection FAILED] 02:32:03
> > domain3.com [Connection OK] 04:02:12
> > ....
> > Can I get this kind of information from the rsyslog server ?
> > Thanks in advance.
> >
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.


_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Information about incoming logs. [ In reply to ]
How can I control this ? Sounds very interesting. Could you please be a little more specific ?

??????? Original Message ???????
On Wednesday, February 17, 2021 11:23 AM, Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com> wrote:

> You can also simply check TCP connections' state. One caveat is that in
> case of rarely-talking sources they might not initiate connection until
> they have data to send.
>
> Mariusz Kruk
> Ekspert ds. Bezpiecze?stwa IT
> COMP S.A.
> Pion Cyberbezpiecze?stwa i Zarz?dzania Ryzykiem
> e-mail: mariusz.kruk@comp.com.pl
> e-mail: mariusz.kruk@safecomp.com
> tel: +48 608 623 299
>
> On 17.02.2021 00:29, David Lang via rsyslog wrote:
>
> > take a look at the impstats module, and the ability for the imtcp
> > module to record per-sender stats
> > if that isn't enough, you can create your own stats via the
> > dyn_stats() functions
> > David Lang
> > On Tue, 16 Feb 2021, odrzen via rsyslog wrote:
> >
> > > Date: Tue, 16 Feb 2021 23:04:40 +0000
> > > From: odrzen via rsyslog rsyslog@lists.adiscon.com
> > > Reply-To: odrzen odrzen@protonmail.com,
> > >     rsyslog-users rsyslog@lists.adiscon.com
> > > To: "rsyslog@lists.adiscon.com" rsyslog@lists.adiscon.com
> > > Cc: odrzen odrzen@protonmail.com
> > > Subject: [rsyslog] Information about incoming logs.
> > > Hello rsyslog community,
> > > I'm new to rsyslog, but have already successfully configured some
> > > machines to send their logs using mutual TLS authentication.
> > > So far so good, but I have some questions :
> > > How can I see from the rsyslog server side how many and which
> > > machines send logs ?
> > > With the command "systemctl status rsyslog" I have a view on what
> > > happens and which machines have problems, but it is not very clear.
> > > Can I somehow see which machines are successfully communicating and
> > > sending their logs ?
> > > Another very interesting information for me, would be to be able to
> > > see the time of the last update for each machine.
> > > For example:
> > > Domain Status Last update
> > > domain1.com [Connection OK] 04:04:33
> > > domain2.com [Connection FAILED] 02:32:03
> > > domain3.com [Connection OK] 04:02:12
> > > ....
> > > Can I get this kind of information from the rsyslog server ?
> > > Thanks in advance.
> > >
> > > rsyslog mailing list
> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> > > POST if you DON'T LIKE THAT.
> >
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> > if you DON'T LIKE THAT.
>
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.


_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Information about incoming logs. [ In reply to ]
I thought there was an option to have the input module keep per-sender stats,
but I'm not finding it now.

with dyn_stats() see
https://www.rsyslog.com/doc/master/configuration/dyn_stats.html

you create a stats type with the dyn_stats() call and you count with the
dyn_inc() call

stats inervals are tied to the interval you set for the impstats, so if you want
per hour or per day stats, it will take post-processing of the stats log

remember, you can log the stats via syslog, at which point you can do all the
filtering that you can do with any other log message

This will not tell you about any errors, but the other rsyslog log messages
should do that.

Simple Event Correlator is a program that you can have watch logs to alert on
specific log messages, or summarize logs over time. Splunk and Elasticsearch are
commonly used for this sort of thing (and having them summarize stats is FAR
more efficient than having them count the number of messages directly)

does this point you in the right direction? if it's still unclear, ask more
questions

David Lang

On Thu, 18 Feb 2021, odrzen wrote:

> Date: Thu, 18 Feb 2021 23:13:36 +0000
> From: odrzen <odrzen@protonmail.com>
> To: David Lang <david@lang.hm>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] Information about incoming logs.
>
> So far, looking for what you are telling us, I have been able to add the following configuration `/etc/rsyslog.conf` :
> ```
> module(load="impstats"
> interval="600"
> severity="7"
> log.syslog="off"
> log.file="/var/log/rsyslog-stats/stats.log")
>
> global(senders.keepTrack="on")
> dyn_stats(name="msg_per_host")
> ```
>
> and here are the results I get every 10 minutes:
> ```
> Fri Feb 19 00:52:08 2021: global: origin=dynstats msg_per_host.ops_overflow=0 msg_per_host.new_metric_add=0 msg_per_host.no_metric=0 msg_per_host.metrics_purged=0 msg_per_host.ops_ignored=0 msg_per_host.purge_triggered=0
> Fri Feb 19 00:52:08 2021: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
> Fri Feb 19 00:52:08 2021: dynafile cache Auditlog: origin=omfile requests=1638 level0=1020 missed=76 evicted=66 maxused=10 closetimeouts=0
> Fri Feb 19 00:52:08 2021: action 0: origin=core.action processed=1638 failed=0 suspended=0 suspended.duration=0 resumed=0
> Fri Feb 19 00:52:08 2021: dynafile cache RemoteLogs: origin=omfile requests=75872 level0=8240 missed=941 evicted=931 maxused=10 closetimeouts=0
> Fri Feb 19 00:52:08 2021: action 1: origin=core.action processed=75872 failed=0 suspended=0 suspended.duration=0 resumed=0
> Fri Feb 19 00:52:08 2021: action 2: origin=core.action processed=74218 failed=0 suspended=0 suspended.duration=0 resumed=0
> Fri Feb 19 00:52:08 2021: action 3: origin=core.action processed=1638 failed=0 suspended=0 suspended.duration=0 resumed=0
> Fri Feb 19 00:52:08 2021: action 4: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Fri Feb 19 00:52:08 2021: action 5: origin=core.action processed=16 failed=0 suspended=0 suspended.duration=0 resumed=0
> Fri Feb 19 00:52:08 2021: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Fri Feb 19 00:52:08 2021: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Fri Feb 19 00:52:08 2021: action 8: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
> Fri Feb 19 00:52:08 2021: msg_per_host: origin=dynstats.bucket
> Fri Feb 19 00:52:08 2021: imudp(*:514): origin=imudp submitted=0
> Fri Feb 19 00:52:08 2021: imudp(*:514): origin=imudp submitted=0
> Fri Feb 19 00:52:08 2021: imtcp(6514): origin=imtcp submitted=75441
> Fri Feb 19 00:52:08 2021: resource-usage: origin=impstats utime=6334580 stime=7795002 maxrss=9256 minflt=2781 majflt=6 inblock=1320 oublock=98680 nvcsw=146787 nivcsw=40
> Fri Feb 19 00:52:08 2021: main Q: origin=core.queue size=0 enqueued=75872 full=0 discarded.full=0 discarded.nf=0 maxqsize=85
> Fri Feb 19 00:52:08 2021: imudp(w0): origin=imudp called.recvmmsg=0 called.recvmsg=0 msgs.received=0
> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_1.com messages=21
> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_2.com messages=21814
> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_3.com messages=5096
> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_4.com messages=25
> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_5.com messages=37
> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_6.com messages=151
> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_7.com messages=13
> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_8.com messages=7460
> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_9.com messages=21
> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_10.com messages=5118
> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_12.com messages=5099
> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_13.com messages=14
> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_14.com messages=7820
> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_15.com messages=123
> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_16.com messages=15170
> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_17.com messages=7459
> ```
>
> About the "the imtcp module to record per-sender stats", I'm not sure I found what you mean, because I only found this parameter in this `dyn_stats(name="msg_per_host")` order.
>
> About dyn_stats(), how can I create my "own" statistics ? For example, can I create reports per/hour or per/days ? In addition, can I include more information in these reports ?
>
> In addition, can I see if there are problems with some machines with connection ( for example wrong certificates ) or bottleneck problems ?
>
>
> Thank you very much for your time. I appreciate your help.
>
>
> ??????? Original Message ???????
> On Wednesday, February 17, 2021 1:29 AM, David Lang <david@lang.hm> wrote:
>
>> take a look at the impstats module, and the ability for the imtcp module to
>> record per-sender stats
>>
>> if that isn't enough, you can create your own stats via the dyn_stats()
>> functions
>>
>> David Lang
>>
>> On Tue, 16 Feb 2021, odrzen via rsyslog wrote:
>>
>>> Date: Tue, 16 Feb 2021 23:04:40 +0000
>>> From: odrzen via rsyslog rsyslog@lists.adiscon.com
>>> Reply-To: odrzen odrzen@protonmail.com,
>>> rsyslog-users rsyslog@lists.adiscon.com
>>> To: "rsyslog@lists.adiscon.com" rsyslog@lists.adiscon.com
>>> Cc: odrzen odrzen@protonmail.com
>>> Subject: [rsyslog] Information about incoming logs.
>>> Hello rsyslog community,
>>> I'm new to rsyslog, but have already successfully configured some machines to send their logs using mutual TLS authentication.
>>> So far so good, but I have some questions :
>>> How can I see from the rsyslog server side how many and which machines send logs ?
>>> With the command "systemctl status rsyslog" I have a view on what happens and which machines have problems, but it is not very clear.
>>> Can I somehow see which machines are successfully communicating and sending their logs ?
>>> Another very interesting information for me, would be to be able to see the time of the last update for each machine.
>>> For example:
>>> Domain Status Last update
>>> domain1.com [Connection OK] 04:04:33
>>> domain2.com [Connection FAILED] 02:32:03
>>> domain3.com [Connection OK] 04:02:12
>>> ....
>>> Can I get this kind of information from the rsyslog server ?
>>> Thanks in advance.
>>>
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Information about incoming logs. [ In reply to ]
If you give each action a unique “name”, then you will see your specified names in the pstats below instead of “action 1” or “action 2” or “action 3".

>> Fri Feb 19 00:52:08 2021: action 1: origin=core.action processed=75872 failed=0 suspended=0 suspended.duration=0 resumed=0
>> Fri Feb 19 00:52:08 2021: action 2: origin=core.action processed=74218 failed=0 suspended=0 suspended.duration=0 resumed=0
>> Fri Feb 19 00:52:08 2021: action 3: origin=core.action processed=1638 failed=0 suspended=0 suspended.duration=0 resumed=0

This makes pstats several orders of magnitude more usable and user friendly. Use the new syntax, and make sure each action has a unique “name=" parameter.

Regards,



> On Feb 18, 2021, at 17:26, David Lang via rsyslog <rsyslog@lists.adiscon.com> wrote:
>
> I thought there was an option to have the input module keep per-sender stats, but I'm not finding it now.
>
> with dyn_stats() see https://www.rsyslog.com/doc/master/configuration/dyn_stats.html
>
> you create a stats type with the dyn_stats() call and you count with the dyn_inc() call
>
> stats inervals are tied to the interval you set for the impstats, so if you want per hour or per day stats, it will take post-processing of the stats log
>
> remember, you can log the stats via syslog, at which point you can do all the filtering that you can do with any other log message
>
> This will not tell you about any errors, but the other rsyslog log messages should do that.
>
> Simple Event Correlator is a program that you can have watch logs to alert on specific log messages, or summarize logs over time. Splunk and Elasticsearch are commonly used for this sort of thing (and having them summarize stats is FAR more efficient than having them count the number of messages directly)
>
> does this point you in the right direction? if it's still unclear, ask more questions
>
> David Lang
>
> On Thu, 18 Feb 2021, odrzen wrote:
>
>> Date: Thu, 18 Feb 2021 23:13:36 +0000
>> From: odrzen <odrzen@protonmail.com>
>> To: David Lang <david@lang.hm>
>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] Information about incoming logs.
>> So far, looking for what you are telling us, I have been able to add the following configuration `/etc/rsyslog.conf` :
>> ```
>> module(load="impstats"
>> interval="600"
>> severity="7"
>> log.syslog="off"
>> log.file="/var/log/rsyslog-stats/stats.log")
>>
>> global(senders.keepTrack="on")
>> dyn_stats(name="msg_per_host")
>> ```
>>
>> and here are the results I get every 10 minutes:
>> ```
>> Fri Feb 19 00:52:08 2021: global: origin=dynstats msg_per_host.ops_overflow=0 msg_per_host.new_metric_add=0 msg_per_host.no_metric=0 msg_per_host.metrics_purged=0 msg_per_host.ops_ignored=0 msg_per_host.purge_triggered=0
>> Fri Feb 19 00:52:08 2021: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
>> Fri Feb 19 00:52:08 2021: dynafile cache Auditlog: origin=omfile requests=1638 level0=1020 missed=76 evicted=66 maxused=10 closetimeouts=0
>> Fri Feb 19 00:52:08 2021: action 0: origin=core.action processed=1638 failed=0 suspended=0 suspended.duration=0 resumed=0
>> Fri Feb 19 00:52:08 2021: dynafile cache RemoteLogs: origin=omfile requests=75872 level0=8240 missed=941 evicted=931 maxused=10 closetimeouts=0
>> Fri Feb 19 00:52:08 2021: action 1: origin=core.action processed=75872 failed=0 suspended=0 suspended.duration=0 resumed=0
>> Fri Feb 19 00:52:08 2021: action 2: origin=core.action processed=74218 failed=0 suspended=0 suspended.duration=0 resumed=0
>> Fri Feb 19 00:52:08 2021: action 3: origin=core.action processed=1638 failed=0 suspended=0 suspended.duration=0 resumed=0
>> Fri Feb 19 00:52:08 2021: action 4: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
>> Fri Feb 19 00:52:08 2021: action 5: origin=core.action processed=16 failed=0 suspended=0 suspended.duration=0 resumed=0
>> Fri Feb 19 00:52:08 2021: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
>> Fri Feb 19 00:52:08 2021: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
>> Fri Feb 19 00:52:08 2021: action 8: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
>> Fri Feb 19 00:52:08 2021: msg_per_host: origin=dynstats.bucket
>> Fri Feb 19 00:52:08 2021: imudp(*:514): origin=imudp submitted=0
>> Fri Feb 19 00:52:08 2021: imudp(*:514): origin=imudp submitted=0
>> Fri Feb 19 00:52:08 2021: imtcp(6514): origin=imtcp submitted=75441
>> Fri Feb 19 00:52:08 2021: resource-usage: origin=impstats utime=6334580 stime=7795002 maxrss=9256 minflt=2781 majflt=6 inblock=1320 oublock=98680 nvcsw=146787 nivcsw=40
>> Fri Feb 19 00:52:08 2021: main Q: origin=core.queue size=0 enqueued=75872 full=0 discarded.full=0 discarded.nf=0 maxqsize=85
>> Fri Feb 19 00:52:08 2021: imudp(w0): origin=imudp called.recvmmsg=0 called.recvmsg=0 msgs.received=0
>> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_1.com messages=21
>> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_2.com messages=21814
>> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_3.com messages=5096
>> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_4.com messages=25
>> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_5.com messages=37
>> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_6.com messages=151
>> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_7.com messages=13
>> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_8.com messages=7460
>> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_9.com messages=21
>> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_10.com messages=5118
>> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_12.com messages=5099
>> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_13.com messages=14
>> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_14.com messages=7820
>> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_15.com messages=123
>> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_16.com messages=15170
>> Fri Feb 19 00:52:08 2021: _sender_stat: sender=example_17.com messages=7459
>> ```
>>
>> About the "the imtcp module to record per-sender stats", I'm not sure I found what you mean, because I only found this parameter in this `dyn_stats(name="msg_per_host")` order.
>>
>> About dyn_stats(), how can I create my "own" statistics ? For example, can I create reports per/hour or per/days ? In addition, can I include more information in these reports ?
>>
>> In addition, can I see if there are problems with some machines with connection ( for example wrong certificates ) or bottleneck problems ?
>>
>>
>> Thank you very much for your time. I appreciate your help.
>>
>>
>> ??????? Original Message ???????
>> On Wednesday, February 17, 2021 1:29 AM, David Lang <david@lang.hm> wrote:
>>
>>> take a look at the impstats module, and the ability for the imtcp module to
>>> record per-sender stats
>>>
>>> if that isn't enough, you can create your own stats via the dyn_stats()
>>> functions
>>>
>>> David Lang
>>>
>>> On Tue, 16 Feb 2021, odrzen via rsyslog wrote:
>>>
>>>> Date: Tue, 16 Feb 2021 23:04:40 +0000
>>>> From: odrzen via rsyslog rsyslog@lists.adiscon.com
>>>> Reply-To: odrzen odrzen@protonmail.com,
>>>> rsyslog-users rsyslog@lists.adiscon.com
>>>> To: "rsyslog@lists.adiscon.com" rsyslog@lists.adiscon.com
>>>> Cc: odrzen odrzen@protonmail.com
>>>> Subject: [rsyslog] Information about incoming logs.
>>>> Hello rsyslog community,
>>>> I'm new to rsyslog, but have already successfully configured some machines to send their logs using mutual TLS authentication.
>>>> So far so good, but I have some questions :
>>>> How can I see from the rsyslog server side how many and which machines send logs ?
>>>> With the command "systemctl status rsyslog" I have a view on what happens and which machines have problems, but it is not very clear.
>>>> Can I somehow see which machines are successfully communicating and sending their logs ?
>>>> Another very interesting information for me, would be to be able to see the time of the last update for each machine.
>>>> For example:
>>>> Domain Status Last update
>>>> domain1.com [Connection OK] 04:04:33
>>>> domain2.com [Connection FAILED] 02:32:03
>>>> domain3.com [Connection OK] 04:02:12
>>>> ....
>>>> Can I get this kind of information from the rsyslog server ?
>>>> Thanks in advance.
>>>>
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
>>
>>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.