Correction - logging has stopped again.
I was watching a tcpdump on the sending server as well as tailing the logs on the receiver and all was fine for about 60 minutes, after which time all logging has stopped again. There is no traffic on either the sending nor receiving servers, all syslog network traffic has completely stopped. The log file on the sending server has not rotated and it continues to populate local to the sending server, but there is no remote logging to the receiver.
The rsyslog process is still running and not throwing any errors. I have restarted the server and the rsyslog service multiple times to no avail.
-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Tobias Heaton via rsyslog
Sent: Tuesday, November 9, 2021 11:17 AM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Tobias Heaton <tobias.heaton@unh.edu>
Subject: Re: [rsyslog] Rsyslog Forward Failure Relate to glib Updates
CAUTION: This email originated from outside of the University System. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Another update: logging has not stopped again for both systems, no interaction with either.
-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Tobias Heaton via rsyslog
Sent: Tuesday, November 9, 2021 10:04 AM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Tobias Heaton <tobias.heaton@unh.edu>
Subject: Re: [rsyslog] Rsyslog Forward Failure Relate to glib Updates
CAUTION: This email originated from outside of the University System. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Update: after performing the downgrade/upgrade again, with subsequent reboots the secondary server is now working properly.
I'm unclear on what caused external logs to stop forwarding during the initial upgrade last month, but at least things are working properly now.
Thank you Mariusz for the quick response - I appreciate it.
Tobias
-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Tobias Heaton via rsyslog
Sent: Tuesday, November 9, 2021 9:22 AM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Tobias Heaton <tobias.heaton@unh.edu>
Subject: Re: [rsyslog] Rsyslog Forward Failure Relate to glib Updates
CAUTION: This email originated from outside of the University System. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hello Mariusz,
I am running rsyslogd 8.24.0-57.0.1.el7_9.1. This version is installed on two identical servers on the same VLAN as the rsyslog receiver.
On 2021-10-18, I upgraded glibc-2.17-324.0.1.el7_9.x86_64 to 2.17-325.0.1.el7_9.x86_64 and rebooted the server. At that time I stopped receiving logs on the syslog receiver from this system. I see no traffic via tcpdump either outbound from the server nor inbound on the syslog receiver.
I downgraded back to glibc-2.17-324.0.1.el7_9.x86_64, rebooted and syslog messages resumed and I could see resumption of UDP traffic. I then re-upgraded glibc back to 2.17-325.0.1.el7_9.x86_64, rebooted and logging seems to be continuing.
As I mentioned I have another, identical server (same architecture, performing the same function as the previous) that has also stopped forwarding logs. I performed the same steps outlined above, but this identical server is still not sending logs. I see no traffic inbound or outbound and I get no error in the rsyslog process - it stops and starts normally.
Local logging is working - this only appears to be affecting log forwarding to another rsyslog receiver.
Tobias
-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz Kruk via rsyslog
Sent: Tuesday, November 9, 2021 9:06 AM
To: rsyslog@lists.adiscon.com
Cc: Mariusz Kruk <kruk@epsilon.eu.org>
Subject: Re: [rsyslog] Rsyslog Forward Failure Relate to glib Updates
CAUTION: This email originated from outside of the University System. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Grub should have nothing to do with it.
Can you be a bit more specific?
1) What is the rsyslog version you're using?
2) What is the glibc version that "works" and what is the version that
"doesn't work".
3) What do you mean by "stopped forwarding"? Does the process itself
work? Does local logging work? Do you see any outbound traffic from rsyslog?
4) Is there anything in rsyslogd own logs?
MK
On 09.11.2021 14:56, Tobias Heaton via rsyslog wrote:
> Good Morning,
>
> I recently found a OEL7 server stopped forwarding configured logs on 2021-10-18.
>
> On that date, I performed Yum updates which included glibc and grub2 packages. Given the coincidence of the updates, I reverted those package updates, rebooted the server and logging suddenly started again.
>
> If I update the glibc and grub2 packages again and reboot, the rsyslog forward logging stop again.
>
> Is there some interoperability problem with the glibc/grub2 package updates and rsyslog?
> _______________________________________________
> rsyslog mailing list
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=04%7C01%7Ctobias.heaton%40unh.edu%7C4914074126284ed75dc108d9a39c6a3a%7Cd6241893512d46dc8d2bbe47e25f5666%7C0%7C0%7C637720715048326977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=I85TbNCu9zoWR39R0MT2Dp3nl%2BHd1AZvJ7NJuGTeTRA%3D&reserved=0
> https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&data=04%7C01%7Ctobias.heaton%40unh.edu%7C4914074126284ed75dc108d9a39c6a3a%7Cd6241893512d46dc8d2bbe47e25f5666%7C0%7C0%7C637720715048326977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=SFCci5GhSV40s%2BlU7IJwgypzOxyt0cT6MpOtm%2BtlzYY%3D&reserved=0
> What's up with rsyslog? Follow https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&data=04%7C01%7Ctobias.heaton%40unh.edu%7C4914074126284ed75dc108d9a39c6a3a%7Cd6241893512d46dc8d2bbe47e25f5666%7C0%7C0%7C637720715048326977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=W8VXY%2FtA3rMGyBwEzOGPcj5t7zmkaCceopXO1dmR3D4%3D&reserved=0
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=04%7C01%7Ctobias.heaton%40unh.edu%7C4914074126284ed75dc108d9a39c6a3a%7Cd6241893512d46dc8d2bbe47e25f5666%7C0%7C0%7C637720715048326977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=I85TbNCu9zoWR39R0MT2Dp3nl%2BHd1AZvJ7NJuGTeTRA%3D&reserved=0 https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&data=04%7C01%7Ctobias.heaton%40unh.edu%7C4914074126284ed75dc108d9a39c6a3a%7Cd6241893512d46dc8d2bbe47e25f5666%7C0%7C0%7C637720715048326977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=SFCci5GhSV40s%2BlU7IJwgypzOxyt0cT6MpOtm%2BtlzYY%3D&reserved=0 What's up with rsyslog? Follow
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&data=04%7C01%7Ctobias.heaton%40unh.edu%7C4914074126284ed75dc108d9a39c6a3a%7Cd6241893512d46dc8d2bbe47e25f5666%7C0%7C0%7C637720715048336973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=fG6ekJ5wZA0Qq1%2FFctXdtEBaC%2F1PNcD1S1NEVXDs7ak%3D&reserved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=04%7C01%7Ctobias.heaton%40unh.edu%7C4914074126284ed75dc108d9a39c6a3a%7Cd6241893512d46dc8d2bbe47e25f5666%7C0%7C0%7C637720715048336973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=wMJT4OawrGLjro5Qqj4pi7TaZUNhqrWlim2bJ9bqD%2FE%3D&reserved=0 https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&data=04%7C01%7Ctobias.heaton%40unh.edu%7C4914074126284ed75dc108d9a39c6a3a%7Cd6241893512d46dc8d2bbe47e25f5666%7C0%7C0%7C637720715048336973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=0kZlDoUyyUkixPIKgdTsxg5H2h4w%2BEkjI4T3Ey1mhsg%3D&reserved=0 What's up with rsyslog? Follow
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&data=04%7C01%7Ctobias.heaton%40unh.edu%7C4914074126284ed75dc108d9a39c6a3a%7Cd6241893512d46dc8d2bbe47e25f5666%7C0%7C0%7C637720715048336973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=fG6ekJ5wZA0Qq1%2FFctXdtEBaC%2F1PNcD1S1NEVXDs7ak%3D&reserved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=04%7C01%7Ctobias.heaton%40unh.edu%7C4914074126284ed75dc108d9a39c6a3a%7Cd6241893512d46dc8d2bbe47e25f5666%7C0%7C0%7C637720715048336973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=wMJT4OawrGLjro5Qqj4pi7TaZUNhqrWlim2bJ9bqD%2FE%3D&reserved=0 https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&data=04%7C01%7Ctobias.heaton%40unh.edu%7C4914074126284ed75dc108d9a39c6a3a%7Cd6241893512d46dc8d2bbe47e25f5666%7C0%7C0%7C637720715048336973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=0kZlDoUyyUkixPIKgdTsxg5H2h4w%2BEkjI4T3Ey1mhsg%3D&reserved=0 What's up with rsyslog? Follow
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&data=04%7C01%7Ctobias.heaton%40unh.edu%7C4914074126284ed75dc108d9a39c6a3a%7Cd6241893512d46dc8d2bbe47e25f5666%7C0%7C0%7C637720715048336973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=fG6ekJ5wZA0Qq1%2FFctXdtEBaC%2F1PNcD1S1NEVXDs7ak%3D&reserved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=04%7C01%7Ctobias.heaton%40unh.edu%7C4914074126284ed75dc108d9a39c6a3a%7Cd6241893512d46dc8d2bbe47e25f5666%7C0%7C0%7C637720715048336973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=wMJT4OawrGLjro5Qqj4pi7TaZUNhqrWlim2bJ9bqD%2FE%3D&reserved=0 https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&data=04%7C01%7Ctobias.heaton%40unh.edu%7C4914074126284ed75dc108d9a39c6a3a%7Cd6241893512d46dc8d2bbe47e25f5666%7C0%7C0%7C637720715048336973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=0kZlDoUyyUkixPIKgdTsxg5H2h4w%2BEkjI4T3Ey1mhsg%3D&reserved=0 What's up with rsyslog? Follow
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&data=04%7C01%7Ctobias.heaton%40unh.edu%7C4914074126284ed75dc108d9a39c6a3a%7Cd6241893512d46dc8d2bbe47e25f5666%7C0%7C0%7C637720715048336973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=fG6ekJ5wZA0Qq1%2FFctXdtEBaC%2F1PNcD1S1NEVXDs7ak%3D&reserved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow
https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.