Hey all,
I'm trying to tackle a long-standing problem we have with timezones. We aim to use UTC everywhere, however we (very) frequently do not have control of some of our sources (Diff departments, diff customers, etc) and frequently receive logs in local time. And often in plain RFC3164 format so it's hard to even tell unless you're watching a source send live.
Historically we've dealt with this problem by requesting the source owners do something on their side, or by using static mapping later on in our collection chain. Both of these are manual and error prone.
I know rsyslog has a wide variety of manipulation options, and I was wondering if _something_ like the following could be done?
- When receiving a message and parsing out the header timestamp, Look at the hour (Call it X)
- Consider the rsyslog receiving system time hour (Call it Y)
- If X and Y differ by more than say 3, rewrite the header timestamp variable to change the hour to Y
- (Standard templates then write the output with the adjusted timestamp)
For the sake of discussion, assume the rsyslog system will always be UTC. Also the reason for saying "more than say 3" is just to account for instances where systems may batch up messages and send hourly...as well as messages that might show up right at say 11:59:59 and just enough system lag breaks the logic. Our usual suspects are more like -07:00 or -09:00 so the precision of this doesn't need to be that tight.
My thinking is if something like this could be done in rsyslog, it would reduce a lot of manual work.
Obviously this wouldn't account for other timestamps that might appear in the messages themselves....one problem at a time. :)
Chris
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
I'm trying to tackle a long-standing problem we have with timezones. We aim to use UTC everywhere, however we (very) frequently do not have control of some of our sources (Diff departments, diff customers, etc) and frequently receive logs in local time. And often in plain RFC3164 format so it's hard to even tell unless you're watching a source send live.
Historically we've dealt with this problem by requesting the source owners do something on their side, or by using static mapping later on in our collection chain. Both of these are manual and error prone.
I know rsyslog has a wide variety of manipulation options, and I was wondering if _something_ like the following could be done?
- When receiving a message and parsing out the header timestamp, Look at the hour (Call it X)
- Consider the rsyslog receiving system time hour (Call it Y)
- If X and Y differ by more than say 3, rewrite the header timestamp variable to change the hour to Y
- (Standard templates then write the output with the adjusted timestamp)
For the sake of discussion, assume the rsyslog system will always be UTC. Also the reason for saying "more than say 3" is just to account for instances where systems may batch up messages and send hourly...as well as messages that might show up right at say 11:59:59 and just enough system lag breaks the logic. Our usual suspects are more like -07:00 or -09:00 so the precision of this doesn't need to be that tight.
My thinking is if something like this could be done in rsyslog, it would reduce a lot of manual work.
Obviously this wouldn't account for other timestamps that might appear in the messages themselves....one problem at a time. :)
Chris
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.