Mailing List Archive

Even if you find a solution for this specific scenario, what will you do
when a third timezone is introduced such that the application log
source, the logging server, and the SIEM are all in different
timezones?  There is simply no graceful to solve this problem.

You can certainly ignore all of the timestamps in all of your events and
only rely on the SIEM ingestion time (as was suggested in an earlier
thread), but otherwise the only way to avoid ambiguity and confusion is
either to force your clients to log in UTC, or force them to supply a
timezone offset in the events they deliver.

The old low resolution timestamp format, i.e. "Jun 10 12:07:41", is a
clear and present danger to accuracy in a global logging environment. 
The lack of the time zone offset can result in events being displaced by
hours (forward or backward), and the lack of the year can result in
events being displaced by a year and some hours (forward or backward).

This is the other reason SIEM engineers will tell you that SIEM
ingestion time is the only time that matters, for it's the only time
value the SIEM has any chance of monitoring reliably.

However, a security investigator will tell you that the original event
time is what's really important because there can be delays in events
reaching the SIEM. Therefore it is still critical to get that original
event time correct.

For the cases of client systems in the same timezone as the logging
server, that are logging in UTC without offset, you could parse rawmsg
to get the original string representation of $timereported, tweak the
format, and append the string "-00:00" to it in a custom output template
for delivery to SIEM.  It's not pretty, but it does work.

This approach doesn't help with other low resolution clients logging
time values in other time zones. For that, as you've discovered, you
have to get even more "creative".

Regards,


On 6/10/20 8:15 AM, Mariusz Kruk via rsyslog wrote:
> Hello there.
>
> I'm fighting with a very annoying issue and ran out of ideas :-)
>
> I'm dealing with a setup in which I forward messages from multiple
> sources to a single consumer (siem/log-management/whatever).
>
> The problem is that the logs come from a variety of solutions some of
> which log in local timezone and some in UTC. Some of them include
> timezone info in the timestamp (and I have no problem with those) but
> some do not.
>
> I convert $timereported from the message to a epoch timestamp and send
> it down the stream along with the source message. If the $timereported
> contains TZ info, everything works great. If it doesn't the conversion
> is done according to /etc/localtime. Which would be quite OK if not
> for some legacy sources reporting in UTC without advertising it in
> source time string.
>
> I thought about just adding a constant offset on a per-source basis.
> But that would work OK... up until next daytime saving change, so I'd
> chave a skew +1->+2 or the other way around. For now it seems the most
> feasible solution though.
>
> I know it'd be best to set TZ to UTC and make the sources report in
> UTC but unfortunately it's not possible.
>
> I've been digging in the docs for last few days but I can't seem to
> find any reasonable solution not involving implementing own DST logic
> in RainerScript (that would be ridiculous :D).
>
> Any pointers where to look for clues? I think I cannot set timezone
> dynamicaly so that rsyslog parses data (or formats time with a given
> TZ offset), right?
>
>
> Best regards,
>
> Mariusz Kruk
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.


_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

one thing that you could do is to have a lookup table that has a time offset for
each host and then when daylight savings changes, update the table and reload
it.

unfortunantly the libc timezone routines make it _really_ hard to do anything
other than convert from the server local timzeone to GMT and back. They have the
routines internally to make it easy, but have repeatedly refused to make them
available to systems, and nobody wants to re-invent the wheel on this (it's a
maintinence nightmare as timezones are a political issue, not a technical one)

And even this gets hard when you have some logs from a machine in the local
timezone and other logs written by apps using GMT

A SEIM does not solve this, it just fakes it, sometimes with better results,
sometimes with worse results.

David Lang


On Wed, 10 Jun 2020, Mariusz Kruk via rsyslog wrote:

> Date: Wed, 10 Jun 2020 15:15:11 +0200
> From: Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
> To: Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
> Cc: Mariusz Kruk <mkr@safecomp.com>
> Subject: [rsyslog] Dynamic timezone?
>
> Hello there.
>
> I'm fighting with a very annoying issue and ran out of ideas :-)
>
> I'm dealing with a setup in which I forward messages from multiple sources to
> a single consumer (siem/log-management/whatever).
>
> The problem is that the logs come from a variety of solutions some of which
> log in local timezone and some in UTC. Some of them include timezone info in
> the timestamp (and I have no problem with those) but some do not.
>
> I convert $timereported from the message to a epoch timestamp and send it
> down the stream along with the source message. If the $timereported contains
> TZ info, everything works great. If it doesn't the conversion is done
> according to /etc/localtime. Which would be quite OK if not for some legacy
> sources reporting in UTC without advertising it in source time string.
>
> I thought about just adding a constant offset on a per-source basis. But that
> would work OK... up until next daytime saving change, so I'd chave a skew
> +1->+2 or the other way around. For now it seems the most feasible solution
> though.
>
> I know it'd be best to set TZ to UTC and make the sources report in UTC but
> unfortunately it's not possible.
>
> I've been digging in the docs for last few days but I can't seem to find any
> reasonable solution not involving implementing own DST logic in RainerScript
> (that would be ridiculous :D).
>
> Any pointers where to look for clues? I think I cannot set timezone
> dynamicaly so that rsyslog parses data (or formats time with a given TZ
> offset), right?
>
>
> Best regards,
>
> Mariusz Kruk
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Firstly - For some strange reason (possibly on my end; will have to
investigate) I'm only receiving a very small subset of list emails. So
I'm responding to responses I read on the web archive of the list.

I fully undersand dangers (or let's say inconviniences) caused by
different time zones. I'd love to have all sources reporting in UTC (but
that's a big nono for local teams which won't hear about working in UTC
- the legacy systems don't understand concept of reporting in different
timezone than presenting timestamps to users) or at least report TZ in
timestamps. As I said, those are some legacy systems which are as they
are and I can't do much about it.

For now I'm doing a lookup (similar to the one suggested by David Lang)
returning an offset specified in seconds and just add the returned
offset to a timestamp. I thought about introducing a "constant" DST
offset so I would add it in case of summer time on top on a per-source
offset but it won't work for different timezones which have different
DST periods so I think I'll just stick to per-source offset lookup.

Thanks for confirming my way of thinking :-)

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.