Mailing List Archive

The queue fills up because rsyslog is not able to deliver the logs fast enough.
You are sending the logs via TCP (encrypted) so the sending is throttled to the
speed that the receiving system can accept them.

Are you using Splunk as the syslog listener to accept the messages?

Splunk is a very poor performing syslog listener, you would be better of running
rsyslog on the splunk system and writing the files to disk there and then
reading those files into Splunk (frequent log rotation and sinkhole options to
have splunk delete the file after it reads it are your friends, you can also
have rsyslog write to different directories based on the sender
hostname/sourcetype/index and have splunk pick those up from the directory
rather than showing the splunk system as the sending host)

David Lang


On Tue, 5 Mar 2024, Andrew Heath via rsyslog wrote:

> Greetings,
> I have a rsyslog server we use to store and forward logs to a Splunk
> instance, however we keep running into the issue where we hit max queue
> size and it writes to disk witch is causing our log partition to fill up, I
> have read the docs and added some more worker threads but to no avail. The
> system has more resources available but for some reason rsyslog is not
> using them to help process and forward logs. I have attached a copy of our
> config file for reference.
>
> [1] https://paste.centos.org/view/36386fa1
>
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Another way of sending to Splunk (if it's indeed the case) is to use
omhttp and send to a HEC input (with the added benefit of the
possibility to add metadata to events).

MK

On 5.03.2024 15:11, David Lang via rsyslog wrote:
> The queue fills up because rsyslog is not able to deliver the logs
> fast enough. You are sending the logs via TCP (encrypted) so the
> sending is throttled to the speed that the receiving system can accept
> them.
>
> Are you using Splunk as the syslog listener to accept the messages?
>
> Splunk is a very poor performing syslog listener, you would be better
> of running rsyslog on the splunk system and writing the files to disk
> there and then reading those files into Splunk (frequent log rotation
> and sinkhole options to have splunk delete the file after it reads it
> are your friends, you can also have rsyslog write to different
> directories based on the sender hostname/sourcetype/index and have
> splunk pick those up from the directory rather than showing the splunk
> system as the sending host)
>
> David Lang
>
>
> On Tue, 5 Mar 2024, Andrew Heath via rsyslog wrote:
>
>> Greetings,
>> I have a rsyslog server we use to store and forward logs to a Splunk
>> instance, however we keep running into the issue where we hit max queue
>> size and it writes to disk witch is causing our log partition to fill
>> up, I
>> have read the docs and added some more worker threads but to no
>> avail. The
>> system has more resources available but for some reason rsyslog is not
>> using them to help process and forward logs. I have attached a copy
>> of our
>> config file for reference.
>>
>> [1] https://paste.centos.org/view/36386fa1
>>
>>
>>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.