Mailing List Archive

Just a note that in practice, CEE is pretty much dead. Pretty much all that
survived is the idea of using JSON to format the data and to use ! to be able to
specify multi-level field names. Everything else in CEE should be treated as an
idea that may or may not be useful rather than a RFC to be followed.

you have pri, which is facility and severity combined, it doesn't make a huge
amount of sense to send all three. In practice the syslog facility and severity
values are what you are going to see.

What I do is to use a standard syslog header, and then in the JSON body I don't
repeat the info that's in the header (although I do like to create a
trusted!relay!foo set of values that populates info to track what systems it's
gone through, doing set $!trusted!relay!lasthop = $!trusted!relay; before
setting the new values)

David Lang

On Sat, 10 Jul 2021, Daniel Pocock via rsyslog wrote:

> Date: Sat, 10 Jul 2021 18:15:36 +0200
> From: Daniel Pocock via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog@lists.adiscon.com
> Cc: Daniel Pocock <daniel@pocock.pro>
> Subject: [rsyslog] CEE field values, ambiguities, sev, syslog!pri, etc
>
>
> Looking at the CEE field names, the descriptions are very brief
>
> For example, what the permitted values for "sev", are they the same as
> the levels in Syslog level?
>
> More confusing are the syslog!pri and syslog!fac fields.
>
> From the Syslog API[1]:
>
> priority = facility | level
>
> example:
>
> pri = LOG_DAEMON | LOG_INFO
> = 0x18 | 6
> = 0x1e
>
>
> but I see that some people are simply putting the level value (e.g. 6)
> into the syslog!pri field.
>
> It would appear more useful to have a syslog!level field for cases where
> we know the level (LOG_INFO) but we might not know the facility value.
>
>
>
> 1. https://www.man7.org/linux/man-pages/man3/syslog.3.html
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

On 10/07/2021 22:38, David Lang wrote:
> Just a note that in practice, CEE is pretty much dead. Pretty much all
> that survived is the idea of using JSON to format the data and to use !
> to be able to specify multi-level field names. Everything else in CEE
> should be treated as an idea that may or may not be useful rather than a
> RFC to be followed.

It is useful enough when trying to normalize logging from a range of
related free software projects

It is useful to use a document that somebody has written rather than
trying to write another one

If anybody knows what the original authors intended for "sev", "pri" and
"syslog!pri", if they ever made an explicit statement in a mailing list,
it would be useful to capture those details. It won't be useful to
everybody but I may put together a small document in the reSIProcate Git
repository about how we are using it with links to any previous
assertions about these fields.

> you have pri, which is facility and severity combined, it doesn't make a
> huge amount of sense to send all three. In practice the syslog facility
> and severity values are what you are going to see.

What I'm doing is putting in the raw values from applications and then
normalizing them to Syslog severity / level values. There is no concept
of Syslog facility in many applications

Therefore, I have values like:
gstreamer!level
resiprocate!level
syslog!level

GStreamer and reSIProcate have more levels than Syslog severity.

However, I also normalize them and send a Syslog severity. I decided to
call it syslog!level - I don't fill in syslog!pri

One other thing to notice is that I'm not capturing any log messages
from regular system daemons. I'm only using this stack to aggregate
messages from SIP and WebRTC. That is why I don't care about using
Syslog facility in this implementation.

If it was a general purpose monitoring setup then I probably would want
to capture the facility value.

> What I do is to use a standard syslog header, and then in the JSON body
> I don't repeat the info that's in the header (although I do like to
> create a trusted!relay!foo set of values that populates info to track
> what systems it's gone through, doing set $!trusted!relay!lasthop =
> $!trusted!relay; before setting the new values)

Thanks, this is helpful feedback. In my case, I've got everything in a
lab environment where I can avoid multiple hops

The type of situation we look at in RTC systems involves something like
this, all the events are triggered by a single client. The client is
making independent connections to the WebSocket (TLS), the Media server
(UDP) and the TURN server (UDP). It is important for developers to
visualize those events in chronological order:


HTTP server: incoming connection
HTTP server: upgrade to WebSocket
SIP server: incoming WebSocket proxied by HTTP server
SIP server: incoming SIP INVITE with SDP
Kurento Media Server: incoming SDP offer
TURN server: incoming STUN request
Kurento Media Server: generate SDP answer
Kurento Media Server: incoming RTP media
Kurento Media Server: incoming RTCP report

Just one phone call can create over 1,000 events, that is why the
severity/level is quite important.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

On Sat, 10 Jul 2021, Daniel Pocock via rsyslog wrote:

> On 10/07/2021 22:38, David Lang wrote:
>> Just a note that in practice, CEE is pretty much dead. Pretty much all
>> that survived is the idea of using JSON to format the data and to use !
>> to be able to specify multi-level field names. Everything else in CEE
>> should be treated as an idea that may or may not be useful rather than a
>> RFC to be followed.
>
> It is useful enough when trying to normalize logging from a range of
> related free software projects
>
> It is useful to use a document that somebody has written rather than
> trying to write another one
>
> If anybody knows what the original authors intended for "sev", "pri" and
> "syslog!pri", if they ever made an explicit statement in a mailing list,
> it would be useful to capture those details. It won't be useful to
> everybody but I may put together a small document in the reSIProcate Git
> repository about how we are using it with links to any previous
> assertions about these fields.
>
>> you have pri, which is facility and severity combined, it doesn't make a
>> huge amount of sense to send all three. In practice the syslog facility
>> and severity values are what you are going to see.
>
> What I'm doing is putting in the raw values from applications and then
> normalizing them to Syslog severity / level values. There is no concept
> of Syslog facility in many applications
>
> Therefore, I have values like:
> gstreamer!level
> resiprocate!level
> syslog!level
>
> GStreamer and reSIProcate have more levels than Syslog severity.
>
> However, I also normalize them and send a Syslog severity. I decided to
> call it syslog!level - I don't fill in syslog!pri
>
> One other thing to notice is that I'm not capturing any log messages
> from regular system daemons. I'm only using this stack to aggregate
> messages from SIP and WebRTC. That is why I don't care about using
> Syslog facility in this implementation.
>
> If it was a general purpose monitoring setup then I probably would want
> to capture the facility value.

facility is fairly worthless nowdays, you should probably put in a localX
facility value, just so that if the logs end up processd by something looking at
it you don't get a '0' (kern) facility.

In practice, I ignore facility and just look at the application name.

>> What I do is to use a standard syslog header, and then in the JSON body
>> I don't repeat the info that's in the header (although I do like to
>> create a trusted!relay!foo set of values that populates info to track
>> what systems it's gone through, doing set $!trusted!relay!lasthop =
>> $!trusted!relay; before setting the new values)
>
> Thanks, this is helpful feedback. In my case, I've got everything in a
> lab environment where I can avoid multiple hops

yeah, in a lab you don't need it, but when you go to a multi-datacenter
geographically disteributed, redundant system environment, you will end up with
cases where one system in the chain misbehaves and capturing the hops makes it
much easier to troubleshoot :-)

> The type of situation we look at in RTC systems involves something like
> this, all the events are triggered by a single client. The client is
> making independent connections to the WebSocket (TLS), the Media server
> (UDP) and the TURN server (UDP). It is important for developers to
> visualize those events in chronological order:
>
>
> HTTP server: incoming connection
> HTTP server: upgrade to WebSocket
> SIP server: incoming WebSocket proxied by HTTP server
> SIP server: incoming SIP INVITE with SDP
> Kurento Media Server: incoming SDP offer
> TURN server: incoming STUN request
> Kurento Media Server: generate SDP answer
> Kurento Media Server: incoming RTP media
> Kurento Media Server: incoming RTCP report
>
> Just one phone call can create over 1,000 events, that is why the
> severity/level is quite important.

yep, make sure you use high precision timestamps and have something in place to
make sure your system clocks are synced. There are various cases where rsyslog
can re-order logs (multi-threaded work under heavy load, logs queued to disk
files, network outages, etc), so the order they appear at the destination is
usually pretty good, but you should not rely on it.

so either high precision timestamps, or a sequence number from the app.

David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

you may want to look and see if there are archives of the project lumberjack
mailing list, that was a wider, slightly less academic, attempt to define a
standard after the CEE effort floundered.

David Lang

On Sat, 10 Jul 2021, David Lang via
rsyslog wrote:

> Date: Sat, 10 Jul 2021 14:22:14 -0700 (PDT)
> From: David Lang via rsyslog <rsyslog@lists.adiscon.com>
> To: Daniel Pocock via rsyslog <rsyslog@lists.adiscon.com>
> Cc: David Lang <david@lang.hm>
> Subject: Re: [rsyslog] CEE field values, ambiguities, sev, syslog!pri, etc
>
> On Sat, 10 Jul 2021, Daniel Pocock via rsyslog wrote:
>
>> On 10/07/2021 22:38, David Lang wrote:
>>> Just a note that in practice, CEE is pretty much dead. Pretty much all
>>> that survived is the idea of using JSON to format the data and to use !
>>> to be able to specify multi-level field names. Everything else in CEE
>>> should be treated as an idea that may or may not be useful rather than a
>>> RFC to be followed.
>>
>> It is useful enough when trying to normalize logging from a range of
>> related free software projects
>>
>> It is useful to use a document that somebody has written rather than
>> trying to write another one
>>
>> If anybody knows what the original authors intended for "sev", "pri" and
>> "syslog!pri", if they ever made an explicit statement in a mailing list,
>> it would be useful to capture those details. It won't be useful to
>> everybody but I may put together a small document in the reSIProcate Git
>> repository about how we are using it with links to any previous
>> assertions about these fields.
>>
>>> you have pri, which is facility and severity combined, it doesn't make a
>>> huge amount of sense to send all three. In practice the syslog facility
>>> and severity values are what you are going to see.
>>
>> What I'm doing is putting in the raw values from applications and then
>> normalizing them to Syslog severity / level values. There is no concept
>> of Syslog facility in many applications
>>
>> Therefore, I have values like:
>> gstreamer!level
>> resiprocate!level
>> syslog!level
>>
>> GStreamer and reSIProcate have more levels than Syslog severity.
>>
>> However, I also normalize them and send a Syslog severity. I decided to
>> call it syslog!level - I don't fill in syslog!pri
>>
>> One other thing to notice is that I'm not capturing any log messages
>> from regular system daemons. I'm only using this stack to aggregate
>> messages from SIP and WebRTC. That is why I don't care about using
>> Syslog facility in this implementation.
>>
>> If it was a general purpose monitoring setup then I probably would want
>> to capture the facility value.
>
> facility is fairly worthless nowdays, you should probably put in a localX
> facility value, just so that if the logs end up processd by something looking
> at it you don't get a '0' (kern) facility.
>
> In practice, I ignore facility and just look at the application name.
>
>>> What I do is to use a standard syslog header, and then in the JSON body
>>> I don't repeat the info that's in the header (although I do like to
>>> create a trusted!relay!foo set of values that populates info to track
>>> what systems it's gone through, doing set $!trusted!relay!lasthop =
>>> $!trusted!relay; before setting the new values)
>>
>> Thanks, this is helpful feedback. In my case, I've got everything in a
>> lab environment where I can avoid multiple hops
>
> yeah, in a lab you don't need it, but when you go to a multi-datacenter
> geographically disteributed, redundant system environment, you will end up
> with cases where one system in the chain misbehaves and capturing the hops
> makes it much easier to troubleshoot :-)
>
>> The type of situation we look at in RTC systems involves something like
>> this, all the events are triggered by a single client. The client is
>> making independent connections to the WebSocket (TLS), the Media server
>> (UDP) and the TURN server (UDP). It is important for developers to
>> visualize those events in chronological order:
>>
>>
>> HTTP server: incoming connection
>> HTTP server: upgrade to WebSocket
>> SIP server: incoming WebSocket proxied by HTTP server
>> SIP server: incoming SIP INVITE with SDP
>> Kurento Media Server: incoming SDP offer
>> TURN server: incoming STUN request
>> Kurento Media Server: generate SDP answer
>> Kurento Media Server: incoming RTP media
>> Kurento Media Server: incoming RTCP report
>>
>> Just one phone call can create over 1,000 events, that is why the
>> severity/level is quite important.
>
> yep, make sure you use high precision timestamps and have something in place
> to make sure your system clocks are synced. There are various cases where
> rsyslog can re-order logs (multi-threaded work under heavy load, logs queued
> to disk files, network outages, etc), so the order they appear at the
> destination is usually pretty good, but you should not rely on it.
>
> so either high precision timestamps, or a sequence number from the app.
>
> David Lang
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.