Mailing List Archive

Daniel,

I'm pretty sure you and I have had at least one yap at some conference or another. Could be I just attended a talk of yours.

I saw your name here and thought "I'm pretty sure I've met him somewhere", and that was somewhat of a pleasant shock, because I've been digging into rsyslog for some stuff I've been thinking about, and it's in a similar vein to what you're talking about here (feeling multi-line data into analytics to help make some sense of it), and frankly it's nice to hear someone else in the same line of work is thinking similar things with respect to these log files (which are chock full of detailed data).

I don't know if what we're after is in fact the same (most folks seem to use logging for error handling, whereas I'm thinking more about gleaning business analytics from the data).

It feels like there's gold in all those log files. It'd be interesting to see how it could be mined.

Regards,

Jim


--
Jim Van Meggelen
ClearlyCore Inc.



+1-416-639-6001 (DID)
+1-877-253-2716 (Canada)
+1-866-644-7729 (USA)
+1-416-425-6111 x6001
jim.vanmeggelen@clearlycore.com
[ http://www.clearlycore.com/ | http://www.clearlycore.com ]

Asterisk: The Definitive Guide
FIFTH EDITION NOW AVAILABLE TO DOWNLOAD:
[ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ]

----- Original Message -----
> From: "Daniel Pocock via rsyslog" <rsyslog@lists.adiscon.com>
> To: rsyslog@lists.adiscon.com
> Cc: "Daniel Pocock" <daniel@pocock.pro>
> Sent: Friday, 9 July, 2021 06:52:16
> Subject: [rsyslog] using Kibana / OpenSearch Dashboards to analyze logs during development

> One of my reasons for looking at rsyslog->OpenSearch integration is for
> the analysis of logs during development of other free software projects,
> especially in the VoIP and WebRTC space.
>
> Here is a quick summary:
>
> - we often have multi-line log messages, such as SIP or SDP message
> bodies, JSON structured logging is very appropriate for these
>
> - modern WebRTC solutions always involve multiple processes (HTTP, TURN,
> SIP, media), so there is a lot of benefit from merging all the logs in
> some sort of server
>
> - we also have some other dedicated tools that can log to OpenSearch,
> for example, the HOMER software[1] captures SIP as well as the RTCP
> reports and lets us visualize them
>
>
> I made some contributions to a few projects (pull requests GStreamer[2],
> Kurento[3], committed in reSIProcate[4]) to log as JSON CEE. This means
> all logs from the whole WebRTC stack can be aggregated and analyzed.
>
> Can anybody make any comments about how to get the best value out of the
> data being captured?
>
> For example, apart from HOMER, are there any reports or front ends to
> ElasticSearch / OpenSearch that are particularly useful in cases like
> this, Syslog data in general?
>
>
>
>
> 1. http://sipcapture.org/
>
> 2. https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/847
>
> 3. https://github.com/Kurento/kurento-media-server/pull/17
>
> 4.
> https://github.com/resiprocate/resiprocate/commit/d1df9aeb956be79253560fe3628b3f37a4ef94f2
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
> THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

multi-line logs are difficult to handle, it would be far easier on you if you
can turn them into single-line logs as early in processing as possible.

There is a lot of business analytics value in logs. the 'easy' way is to throw
it into Splunk or ElasticSearch and depend on queries there, but that ends up
being rather inefficient. I like to get the logs into those tools to make them
easy to explore, but once you figure out what you want to know you can be far
more efficient in the gathering of your metrics.

you can use something like Simple Event Correlator to turn a series of events
into counts that you can then graph, and once you have graphable numbers, then
something like the holt-winters algorithm that RRDtool implements can predict
normal values and alert you when you stray (and the beauty of holt-winters is
that the same numerical value can produce a 'unexpecteedly high' alert at 3am
sunday morning, 'unexpectedly low' at 10am monday, and be in the normal range at
3pm on monday)

Rsyslog is not an analysis engine, but it's a very good routing/reformating
engine for single-line logs (it can do some handling of multi-line logs, but
that tends to just push the failure down to the next component)

One thing to remember is that rsyslog is a 'best effort' logging, there are ways
to make it handle failures, but there remain failures that can cause logs to be
lost. Don't use rsyslog as the only path for content that will cost you money
if it's lost.

https://www.usenix.org/publications/login/david-lang-series
https://www.usenix.org/publications/login/april14/lang
https://www.usenix.org/conference/lisa12/technical-sessions/presentation/lang_david
http://ristov.users.sourceforge.net/publications/cogsima15-sec-web.pdf

David Lang



On Fri, 9 Jul 2021, Jim Van Meggelen via rsyslog
wrote:

> Date: Fri, 9 Jul 2021 07:42:28 -0500 (CDT)
> From: Jim Van Meggelen via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Jim Van Meggelen <jim.vanmeggelen@clearlycore.com>
> Subject: Re: [rsyslog] using Kibana / OpenSearch Dashboards to analyze logs
> during development
>
> Daniel,
>
> I'm pretty sure you and I have had at least one yap at some conference or another. Could be I just attended a talk of yours.
>
> I saw your name here and thought "I'm pretty sure I've met him somewhere", and that was somewhat of a pleasant shock, because I've been digging into rsyslog for some stuff I've been thinking about, and it's in a similar vein to what you're talking about here (feeling multi-line data into analytics to help make some sense of it), and frankly it's nice to hear someone else in the same line of work is thinking similar things with respect to these log files (which are chock full of detailed data).
>
> I don't know if what we're after is in fact the same (most folks seem to use logging for error handling, whereas I'm thinking more about gleaning business analytics from the data).
>
> It feels like there's gold in all those log files. It'd be interesting to see how it could be mined.
>
> Regards,
>
> Jim
>
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

also, dig into mmnormalize (liblognorm), it's a very efficient parse engine for
extracting values out of logs. The Dyn_stats() feature in rsyslog ends up being
a rather powerful tool for summarizing things (SEC is more powerful, but you can
do a lot with just dyn_stats() )

David Lang

On Fri, 9 Jul 2021, Jim Van Meggelen via rsyslog wrote:

> Date: Fri, 9 Jul 2021 07:42:28 -0500 (CDT)
> From: Jim Van Meggelen via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Jim Van Meggelen <jim.vanmeggelen@clearlycore.com>
> Subject: Re: [rsyslog] using Kibana / OpenSearch Dashboards to analyze logs
> during development
>
> Daniel,
>
> I'm pretty sure you and I have had at least one yap at some conference or another. Could be I just attended a talk of yours.
>
> I saw your name here and thought "I'm pretty sure I've met him somewhere", and that was somewhat of a pleasant shock, because I've been digging into rsyslog for some stuff I've been thinking about, and it's in a similar vein to what you're talking about here (feeling multi-line data into analytics to help make some sense of it), and frankly it's nice to hear someone else in the same line of work is thinking similar things with respect to these log files (which are chock full of detailed data).
>
> I don't know if what we're after is in fact the same (most folks seem to use logging for error handling, whereas I'm thinking more about gleaning business analytics from the data).
>
> It feels like there's gold in all those log files. It'd be interesting to see how it could be mined.
>
> Regards,
>
> Jim
>
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

here is a paper on the parsing performance
https://www.fernuni-hagen.de/rechnerarchitektur/docs/rainer_gerhards.pdf

On Fri, 9 Jul 2021, David Lang via rsyslog wrote:

> Date: Fri, 9 Jul 2021 15:40:19 -0700 (PDT)
> From: David Lang via rsyslog <rsyslog@lists.adiscon.com>
> To: Jim Van Meggelen via rsyslog <rsyslog@lists.adiscon.com>
> Cc: David Lang <david@lang.hm>
> Subject: Re: [rsyslog] using Kibana / OpenSearch Dashboards to analyze logs
> during development
>
> also, dig into mmnormalize (liblognorm), it's a very efficient parse engine
> for extracting values out of logs. The Dyn_stats() feature in rsyslog ends up
> being a rather powerful tool for summarizing things (SEC is more powerful,
> but you can do a lot with just dyn_stats() )
>
> David Lang
>
> On Fri, 9 Jul 2021, Jim Van Meggelen via rsyslog wrote:
>
>> Date: Fri, 9 Jul 2021 07:42:28 -0500 (CDT)
>> From: Jim Van Meggelen via rsyslog <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: Jim Van Meggelen <jim.vanmeggelen@clearlycore.com>
>> Subject: Re: [rsyslog] using Kibana / OpenSearch Dashboards to analyze logs
>> during development
>>
>> Daniel,
>>
>> I'm pretty sure you and I have had at least one yap at some conference or
>> another. Could be I just attended a talk of yours.
>>
>> I saw your name here and thought "I'm pretty sure I've met him somewhere",
>> and that was somewhat of a pleasant shock, because I've been digging into
>> rsyslog for some stuff I've been thinking about, and it's in a similar vein
>> to what you're talking about here (feeling multi-line data into analytics
>> to help make some sense of it), and frankly it's nice to hear someone else
>> in the same line of work is thinking similar things with respect to these
>> log files (which are chock full of detailed data).
>>
>> I don't know if what we're after is in fact the same (most folks seem to
>> use logging for error handling, whereas I'm thinking more about gleaning
>> business analytics from the data).
>>
>> It feels like there's gold in all those log files. It'd be interesting to
>> see how it could be mined.
>>
>> Regards,
>>
>> Jim
>>
>>
>>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

David,

Thank you for your generosity in taking the time to write out such a thoughtful response. Wow.

You have given me some really interesting ideas to pursue. I am grateful.

I am going to chew on this for a bit. I have some reading to do (including a few articles written by you!).

Thanks again, and very warm regards,

Jim


--
Jim Van Meggelen
ClearlyCore Inc.



+1-416-639-6001 (DID)
+1-877-253-2716 (Canada)
+1-866-644-7729 (USA)
+1-416-425-6111 x6001
jim.vanmeggelen@clearlycore.com
[ http://www.clearlycore.com/ | http://www.clearlycore.com ]

Asterisk: The Definitive Guide
FIFTH EDITION NOW AVAILABLE TO DOWNLOAD:
[ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ]

----- Original Message -----
> From: "David Lang" <david@lang.hm>
> To: "rsyslog-users" <rsyslog@lists.adiscon.com>
> Cc: "Jim Van Meggelen" <jim.vanmeggelen@clearlycore.com>
> Sent: Friday, 9 July, 2021 18:38:08
> Subject: Re: [rsyslog] using Kibana / OpenSearch Dashboards to analyze logs during development

> multi-line logs are difficult to handle, it would be far easier on you if you
> can turn them into single-line logs as early in processing as possible.
>
> There is a lot of business analytics value in logs. the 'easy' way is to throw
> it into Splunk or ElasticSearch and depend on queries there, but that ends up
> being rather inefficient. I like to get the logs into those tools to make them
> easy to explore, but once you figure out what you want to know you can be far
> more efficient in the gathering of your metrics.
>
> you can use something like Simple Event Correlator to turn a series of events
> into counts that you can then graph, and once you have graphable numbers, then
> something like the holt-winters algorithm that RRDtool implements can predict
> normal values and alert you when you stray (and the beauty of holt-winters is
> that the same numerical value can produce a 'unexpecteedly high' alert at 3am
> sunday morning, 'unexpectedly low' at 10am monday, and be in the normal range at
> 3pm on monday)
>
> Rsyslog is not an analysis engine, but it's a very good routing/reformating
> engine for single-line logs (it can do some handling of multi-line logs, but
> that tends to just push the failure down to the next component)
>
> One thing to remember is that rsyslog is a 'best effort' logging, there are ways
> to make it handle failures, but there remain failures that can cause logs to be
> lost. Don't use rsyslog as the only path for content that will cost you money
> if it's lost.
>
> https://www.usenix.org/publications/login/david-lang-series
> https://www.usenix.org/publications/login/april14/lang
> https://www.usenix.org/conference/lisa12/technical-sessions/presentation/lang_david
> http://ristov.users.sourceforge.net/publications/cogsima15-sec-web.pdf
>
> David Lang
>
>
>
> On Fri, 9 Jul 2021, Jim Van Meggelen via rsyslog
> wrote:
>
>> Date: Fri, 9 Jul 2021 07:42:28 -0500 (CDT)
>> From: Jim Van Meggelen via rsyslog <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: Jim Van Meggelen <jim.vanmeggelen@clearlycore.com>
>> Subject: Re: [rsyslog] using Kibana / OpenSearch Dashboards to analyze logs
>> during development
>>
>> Daniel,
>>
>> I'm pretty sure you and I have had at least one yap at some conference or
>> another. Could be I just attended a talk of yours.
>>
>> I saw your name here and thought "I'm pretty sure I've met him somewhere", and
>> that was somewhat of a pleasant shock, because I've been digging into rsyslog
>> for some stuff I've been thinking about, and it's in a similar vein to what
>> you're talking about here (feeling multi-line data into analytics to help make
>> some sense of it), and frankly it's nice to hear someone else in the same line
>> of work is thinking similar things with respect to these log files (which are
>> chock full of detailed data).
>>
>> I don't know if what we're after is in fact the same (most folks seem to use
>> logging for error handling, whereas I'm thinking more about gleaning business
>> analytics from the data).
>>
>> It feels like there's gold in all those log files. It'd be interesting to see
>> how it could be mined.
>>
>> Regards,
>>
>> Jim
>>
>>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.