Mailing List Archive

MaxMessageSize makes rsyslog stop logging after a few minutes
Hi,

We have a setup with 10 satellite rsyslog servers that receive the logs
from all our application/infra servers and then sends all this logs to a
central rsyslog server.

We and average of 115k requests per 10seconds arriving on our central
rsyslog server.

One of our application owners asked us to increase the MaxMessageSize on
the application servers rsyslog and on the satellite and central rsyslog
servers in order to accommodate for some of the application big log
messages that were being truncated by rsyslog.

The biggest log we found on the application was around 1m, so we defined
$MaxMessageSize 1.5m on the application servers rsyslog and the same on
the satellite and central rsyslog servers.

After we activated this everything looked fine, but after 5 minutes the
central rsyslog server stopped logging and the satellite rsyslog servers
started spooling their messages to disk as they couldn't send them to
the central rsyslog server.

The only error we could see on the central rsyslog logs was this:

"... netstream session ... from ... will be closed due to error
[v8.24.0-52.el7 try http://www.rsyslog.com/e/2165]"

From documentation:
rsyslog error 2165

Error occured during socket rcv operation. This error code indicates
that there was a problem receiving from the remote host. Check network
connectivity to see if there is a physical problem. This can especially
occur if there is a partly-functioning network component.

After restarting rsyslog on central server it starts logging again, but
after 5 minutes stops logging again.

So for now we had to remove the MaxMessageSize parameter from the
central rsyslog.

On the satellite rsyslog servers and application servers rsyslog the
parameter is still there and has not caused any issue.


Anyone has ever experienced such an issue? Any ideas?


Best regards,
Ricardo Esteves.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: MaxMessageSize makes rsyslog stop logging after a few minutes [ In reply to ]
You are using a version of rsyslog that is roughly five years old. In the meantime, MUCH repair and optimization work has been done. You should first and foremost upgrade rsyslog itself, then the list may be able to help resolve the issue if it persists.

Regards,


> On Mar 24, 2022, at 05:45, Ricardo Esteves via rsyslog <rsyslog@lists.adiscon.com> wrote:
>
> Hi,
>
> We have a setup with 10 satellite rsyslog servers that receive the logs from all our application/infra servers and then sends all this logs to a central rsyslog server.
>
> We and average of 115k requests per 10seconds arriving on our central rsyslog server.
>
> One of our application owners asked us to increase the MaxMessageSize on the application servers rsyslog and on the satellite and central rsyslog servers in order to accommodate for some of the application big log messages that were being truncated by rsyslog.
>
> The biggest log we found on the application was around 1m, so we defined $MaxMessageSize 1.5m on the application servers rsyslog and the same on the satellite and central rsyslog servers.
>
> After we activated this everything looked fine, but after 5 minutes the central rsyslog server stopped logging and the satellite rsyslog servers started spooling their messages to disk as they couldn't send them to the central rsyslog server.
>
> The only error we could see on the central rsyslog logs was this:
>
> "... netstream session ... from ... will be closed due to error [v8.24.0-52.el7 try http://www.rsyslog.com/e/2165]"
>
> From documentation:
> rsyslog error 2165
>
> Error occured during socket rcv operation. This error code indicates that there was a problem receiving from the remote host. Check network connectivity to see if there is a physical problem. This can especially occur if there is a partly-functioning network component.
>
> After restarting rsyslog on central server it starts logging again, but after 5 minutes stops logging again.
>
> So for now we had to remove the MaxMessageSize parameter from the central rsyslog.
>
> On the satellite rsyslog servers and application servers rsyslog the parameter is still there and has not caused any issue.
>
>
> Anyone has ever experienced such an issue? Any ideas?
>
>
> Best regards,
> Ricardo Esteves.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: MaxMessageSize makes rsyslog stop logging after a few minutes [ In reply to ]
The size type parameters only support integer sizes, so it be 1m or 2m
but not 1.5m - 1536k will do it. Not sure if it is related, and you
should probably see a startup error in rsyslog's log.

HTH
Rainer

El jue, 24 mar 2022 a las 11:46, Ricardo Esteves via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> Hi,
>
> We have a setup with 10 satellite rsyslog servers that receive the logs
> from all our application/infra servers and then sends all this logs to a
> central rsyslog server.
>
> We and average of 115k requests per 10seconds arriving on our central
> rsyslog server.
>
> One of our application owners asked us to increase the MaxMessageSize on
> the application servers rsyslog and on the satellite and central rsyslog
> servers in order to accommodate for some of the application big log
> messages that were being truncated by rsyslog.
>
> The biggest log we found on the application was around 1m, so we defined
> $MaxMessageSize 1.5m on the application servers rsyslog and the same on
> the satellite and central rsyslog servers.
>
> After we activated this everything looked fine, but after 5 minutes the
> central rsyslog server stopped logging and the satellite rsyslog servers
> started spooling their messages to disk as they couldn't send them to
> the central rsyslog server.
>
> The only error we could see on the central rsyslog logs was this:
>
> "... netstream session ... from ... will be closed due to error
> [v8.24.0-52.el7 try http://www.rsyslog.com/e/2165]"
>
> From documentation:
> rsyslog error 2165
>
> Error occured during socket rcv operation. This error code indicates
> that there was a problem receiving from the remote host. Check network
> connectivity to see if there is a physical problem. This can especially
> occur if there is a partly-functioning network component.
>
> After restarting rsyslog on central server it starts logging again, but
> after 5 minutes stops logging again.
>
> So for now we had to remove the MaxMessageSize parameter from the
> central rsyslog.
>
> On the satellite rsyslog servers and application servers rsyslog the
> parameter is still there and has not caused any issue.
>
>
> Anyone has ever experienced such an issue? Any ideas?
>
>
> Best regards,
> Ricardo Esteves.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.