Mailing List Archive: rsyslog omkafka performance tuning

Hi,

I'm hoping to get some help tuning my rsyslog configuration for
performance. I'm running rsyslog 8.2302.0 using the omkafka plugin on a set
of kubernetes pods.
However, just examining one pod, it feels like something is not configured
properly (see config below). What I'm trying to do is use disk-assisted
queues
so that if the queue is full or a kubernetes pod gets deleted, then the pod
should save the data to disk so that it can be processed when the next
kubernetes pod comes back up.

For one pod, I see it is writing about 150 msg/sec on average. However,
when a spike of incoming data came in, the rsyslog service started quickly
falling behind resulting in messages being written to disk.
I restarted the pod and we were no longer receiving as much input traffic
so when that pod came back up, it processed 2.6 million messages that were
on disk for about 15 minutes -- about 3000 msg/sec.

My question is this: Why was this faster to crank through all the messages
on disk than it would be when messages are just coming through? Is there
something I can do to tweak my config
to make this perform better?

Note: kafka cluster is in the US region and rsyslog-service is in the EU
region so I know there will be some latency there but not really sure how
to get that metric to see what that is.

Thanks,
-Ken

Rsyslog config:

module(load="omkafka")
module(load="imptcp" threads="2" ProcessOnPoller="off")
module(load="omprog")

module(
load="impstats"
interval="10"
format="json"
resetCounters="off"
ruleset="process_stats"
)

input(type="imptcp" port="1514" ruleset="writeToKafka" KeepAlive="on"
DiscardTruncatedMsg="on")

template(name="raw" type="string" string="%rawmsg%")

ruleset(name="writeToKafka") {
action(type="omkafka"
confParam=["socket.keepalive.enable=true",
"compression.codec=none",
"batch.num.messages=20000",
"statistics.interval.ms=15000",
"queue.buffering.max.ms=100",
"queue.buffering.max.messages=200000",
"security.protocol=SASL_SSL",
"sasl.mechanisms=PLAIN",
"sasl.username=<username>",
"sasl.password=<password>"]
topic="rsyslog-ingest"
broker="<kafkaBroker>"
template="raw"
partitions.auto="on"
resubmitonfailure="on"
keepfailedmessages="on"
failedmsgfile="/data/failed-messages.log"
action.resumeRetryCount="-1"
action.resumeInterval="10"
queue.type="LinkedList"
queue.filename="remote-kafka"
queue.spoolDirectory="/data"
queue.saveonshutdown="on"
queue.size="250000"
queue.highwatermark="225000"
queue.lowwatermark="200000"
queue.maxdiskspace="7g"
queue.maxfilesize="10m"
queue.discardseverity="8"
errorFile="/var/log/rsyslog-kafka-errors.log"
)
}

ruleset(name="process_stats") {
action(
type="omprog"
name="to_exporter"
binary="/usr/local/bin/rsyslog_exporter"
)
}
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

I see you have pstats enabled, what does it show?

also, during the overload time, what does top show? (hit H to show threads so
that we can see per-thread data)

this would be the first time I have ever seen messages from disk processed
FASTER than new messages, normally messages from disk are horribly slow in
comparison

Is this the entire config? nothing else happening?

I am not familiar with the kafka output module, so I don't know if it supports
bulk mode processing (processing multiple log messages as a batch). From the
config parameters it would look like it should. The default for rsyslog is to
handle batches of 128 messages IIRC, but it wouldn't hurt to try setting the
batch size to a larger value (since you seem to be setting it to 2000 for kafka)

also, please look at logs from the kafka process. 150 logs/sec is only ~1
batch/sec and that indicates a serious problem communicating with kafka.

David Lang

On Thu, 11 May 2023, Kenneth Logan via rsyslog wrote:

> Date: Thu, 11 May 2023 19:55:08 -0500
> From: Kenneth Logan via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog@lists.adiscon.com
> Cc: Kenneth Logan <noobguru@gmail.com>
> Subject: [rsyslog] rsyslog omkafka performance tuning
>
> Hi,
>
> I'm hoping to get some help tuning my rsyslog configuration for
> performance. I'm running rsyslog 8.2302.0 using the omkafka plugin on a set
> of kubernetes pods.
> However, just examining one pod, it feels like something is not configured
> properly (see config below). What I'm trying to do is use disk-assisted
> queues
> so that if the queue is full or a kubernetes pod gets deleted, then the pod
> should save the data to disk so that it can be processed when the next
> kubernetes pod comes back up.
>
> For one pod, I see it is writing about 150 msg/sec on average. However,
> when a spike of incoming data came in, the rsyslog service started quickly
> falling behind resulting in messages being written to disk.
> I restarted the pod and we were no longer receiving as much input traffic
> so when that pod came back up, it processed 2.6 million messages that were
> on disk for about 15 minutes -- about 3000 msg/sec.
>
> My question is this: Why was this faster to crank through all the messages
> on disk than it would be when messages are just coming through? Is there
> something I can do to tweak my config
> to make this perform better?
>
> Note: kafka cluster is in the US region and rsyslog-service is in the EU
> region so I know there will be some latency there but not really sure how
> to get that metric to see what that is.
>
> Thanks,
> -Ken
>
> Rsyslog config:
>
> module(load="omkafka")
> module(load="imptcp" threads="2" ProcessOnPoller="off")
> module(load="omprog")
>
> module(
> load="impstats"
> interval="10"
> format="json"
> resetCounters="off"
> ruleset="process_stats"
> )
>
> input(type="imptcp" port="1514" ruleset="writeToKafka" KeepAlive="on"
> DiscardTruncatedMsg="on")
>
> template(name="raw" type="string" string="%rawmsg%")
>
> ruleset(name="writeToKafka") {
> action(type="omkafka"
> confParam=[."socket.keepalive.enable=true",
> "compression.codec=none",
> "batch.num.messages=20000",
> "statistics.interval.ms=15000",
> "queue.buffering.max.ms=100",
> "queue.buffering.max.messages=200000",
> "security.protocol=SASL_SSL",
> "sasl.mechanisms=PLAIN",
> "sasl.username=<username>",
> "sasl.password=<password>"]
> topic="rsyslog-ingest"
> broker="<kafkaBroker>"
> template="raw"
> partitions.auto="on"
> resubmitonfailure="on"
> keepfailedmessages="on"
> failedmsgfile="/data/failed-messages.log"
> action.resumeRetryCount="-1"
> action.resumeInterval="10"
> queue.type="LinkedList"
> queue.filename="remote-kafka"
> queue.spoolDirectory="/data"
> queue.saveonshutdown="on"
> queue.size="250000"
> queue.highwatermark="225000"
> queue.lowwatermark="200000"
> queue.maxdiskspace="7g"
> queue.maxfilesize="10m"
> queue.discardseverity="8"
> errorFile="/var/log/rsyslog-kafka-errors.log"
> )
> }
>
> ruleset(name="process_stats") {
> action(
> type="omprog"
> name="to_exporter"
> binary="/usr/local/bin/rsyslog_exporter"
> )
> }
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.