Mailing List Archive

when you get into this mode, run top and hit H to show per-thread cpu usage

does it show the imtcp thread using 100% cpu? if so, you may have run into a
problem that I am occasionally having.

does tis happen during the time of your debug log?

David Lang

On Mon, 24 Aug 2020, Tod A Sandman via rsyslog wrote:

> Date: Mon, 24 Aug 2020 21:30:49 -0500
> From: Tod A Sandman via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog@lists.adiscon.com
> Cc: Tod A Sandman <sandmant@rice.edu>
> Subject: [rsyslog] Palo Alto Network device logging stopped working
>
> I am running a central log server with rsyslog-8.2006 and the latest release of RHEL7.
>
> A few weeks ago logging stopped working for one type of client, our security group's (ISO) Palo Alto Network (PAN) devices. These log to a dedicated port (9022) using SSL and imtcp. I've tried both the ossl and gtls drivers and get the same result: tcpdump shows network traffic from the clients and lsof shows established connections, but no logging is taking place. Logging via the same port/config works fine for Linux rsyslog clients I've tested. It sure seems like a client or a network issue to me:
>
> imtcp.c : nsd_ossl.c: osslRecordRecv: Errno 104, connection resetted by peer
>
>
> But our ISO group is at a loss and I have no more ideas. I'm hoping that someone can look at my config and error logs and confirm this is a client issue and/or give me any suggestions for further debugging.
>
> To minimize the configuration debugging and noise, today I had the ISO folk point a single PAN device to a standby rsyslog server identical to the production server. I then removed all client logging configuration on the standby except for the PAN client configuration.
>
> Attached (rsyslog_debug_fail.log.gz) is a debug log after starting rsyslog, waiting a few minutes, and stopping rsyslog.
>
> Attached also are my config files.
>
> Thanks for any advice.
>
>
> Tod A. Sandman
> Office of Information Technology
> Rice University
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

I suspect they have a problem with their framing (rings a bell). Can
you post a tcpdump of a few messages?

Rainer

El mar., 25 ago. 2020 a las 4:30, Tod A Sandman via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> I am running a central log server with rsyslog-8.2006 and the latest release of RHEL7.
>
> A few weeks ago logging stopped working for one type of client, our security group's (ISO) Palo Alto Network (PAN) devices. These log to a dedicated port (9022) using SSL and imtcp. I've tried both the ossl and gtls drivers and get the same result: tcpdump shows network traffic from the clients and lsof shows established connections, but no logging is taking place. Logging via the same port/config works fine for Linux rsyslog clients I've tested. It sure seems like a client or a network issue to me:
>
> imtcp.c : nsd_ossl.c: osslRecordRecv: Errno 104, connection resetted by peer
>
>
> But our ISO group is at a loss and I have no more ideas. I'm hoping that someone can look at my config and error logs and confirm this is a client issue and/or give me any suggestions for further debugging.
>
> To minimize the configuration debugging and noise, today I had the ISO folk point a single PAN device to a standby rsyslog server identical to the production server. I then removed all client logging configuration on the standby except for the PAN client configuration.
>
> Attached (rsyslog_debug_fail.log.gz) is a debug log after starting rsyslog, waiting a few minutes, and stopping rsyslog.
>
> Attached also are my config files.
>
> Thanks for any advice.
>
>
> Tod A. Sandman
> Office of Information Technology
> Rice University
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

> does it show the imtcp thread using 100% cpu? if so, you may have run into a problem that I am occasionally having.

No, not at all. Not on my standby which I have isolated and am using to debug this PAN issue.

However, speak of the devil, I went and checked my production box and it is exactly as you say. And, Proofpoint logging has indeed stopped (last night at 11:24pmCST). Note that Proofpoint is the only other source that uses imtcp. The bulk of the sources are non-SSL and use imptcp. And they never seem to have any trouble.

I have run into issues over the last few months with Proofpoint and/or PAN logging stopping. In an attempt to debug, I pointed Proofpoint to our standby server over a month ago, and the 100% CPU issue ended and all logging worked. Within a few days, the PAN logging stopped, but the 100% CPU did not take place. Proofpoint logging to the standby continued. Yesterday I pointed Proofpoint back to our production server so I could test PAN on the standby, and Proofpoint logging worked fine until last night.

I have just blocked via iptables connections from the PAN devices to our primary (not working anyway) and have restarted rsyslog. Proofpoint logs are flowing again.

I will focus now on the PAN devices and our standby server.

This does not make a lot of sense yet. It seems:

PAN and Proofpoint logging together led to the 100% CPU issue, and both logs stopped.

Proofpoint logging alone worked fine (though I do see some ossl syslog error messages similar to what I see for PAN).

PAN logging (no Proofpoint) stopped; no CPU load issue. Can't even get it working on standby, rsyslog restarts, etc.


Tod A. Sandman
Office of Information Technology
Rice University
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

On 2020-08-25 07:18, Tod A Sandman via rsyslog wrote:
>> does it show the imtcp thread using 100% cpu? if so, you may have run
>> into a problem that I am occasionally having.
>
> No, not at all. Not on my standby which I have isolated and am using
> to debug this PAN issue.
>
> However, speak of the devil, I went and checked my production box and
> it is exactly as you say. And, Proofpoint logging has indeed stopped
> (last night at 11:24pmCST). Note that Proofpoint is the only other
> source that uses imtcp. The bulk of the sources are non-SSL and use
> imptcp. And they never seem to have any trouble.
>
> I have run into issues over the last few months with Proofpoint and/or
> PAN logging stopping. In an attempt to debug, I pointed Proofpoint to
> our standby server over a month ago, and the 100% CPU issue ended and
> all logging worked. Within a few days, the PAN logging stopped, but
> the 100% CPU did not take place. Proofpoint logging to the standby
> continued. Yesterday I pointed Proofpoint back to our production
> server so I could test PAN on the standby, and Proofpoint logging
> worked fine until last night.
>
> I have just blocked via iptables connections from the PAN devices to
> our primary (not working anyway) and have restarted rsyslog.
> Proofpoint logs are flowing again.
>
> I will focus now on the PAN devices and our standby server.
>
> This does not make a lot of sense yet. It seems:
>
> PAN and Proofpoint logging together led to the 100% CPU issue, and
> both logs stopped.
>
> Proofpoint logging alone worked fine (though I do see some ossl syslog
> error messages similar to what I see for PAN).
>
> PAN logging (no Proofpoint) stopped; no CPU load issue. Can't even
> get it working on standby, rsyslog restarts, etc.
>
>
> Tod A. Sandman
> Office of Information Technology
> Rice University
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.

Upgrade your PA...this is a bug.

James
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

> I suspect they have a problem with their framing (rings a bell). Can you post a tcpdump of a few messages?

I have attached a tcpudmp.


Tod A. Sandman
Office of Information Technology
Rice University