Good day
I've configured the omkafka module to push logs to Kafka and added a disk-assisted queue configuration to this action. The idea being, that if the Kafka cluster is temporarily unavailable, logs will be queued and prevent message loss. It should also be noted that this whole Kafka configuration is in its own ruleset and called from other rulesets in order to output data to Kafka.
The issue I'm having is that the queue doesn't hold onto messages and simply passes it on (I can see in the relevant statistics that the queue processed the messages) even when the Kafka cluster can't be reached and the action suspends. The librdkafka library - used by omkafka - also has its own queue, but this fills up and message loss occurs. I've also tried to rather set the queue on the ruleset itself, but that too makes no difference.
Below is the queue configuration within the action(type="omkafka" ...) definition.
# Queue Settings
queue.type="LinkedList" # In-memory queue with dynamic memory allocation
queue.spoolDirectory="/var/spool/rsyslog" # Path for disk-assisted queue
queue.size="65000" # Total queue size
queue.highwatermark="60000" # Write messages to disk if highwatermark breached
queue.lowwatermark="50000" # Use in-memory queue again below this mark
queue.filename="kafka_output" # Enables disk-assisted queue and names it
queue.maxdiskspace="18g" # Maximum disk space disk-queue can use
queue.saveonshutdown="on" # Save queue contents to disk on shutdown
Any assistance in this regard would be greatly appreciated.
Kind Regards
---
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
I've configured the omkafka module to push logs to Kafka and added a disk-assisted queue configuration to this action. The idea being, that if the Kafka cluster is temporarily unavailable, logs will be queued and prevent message loss. It should also be noted that this whole Kafka configuration is in its own ruleset and called from other rulesets in order to output data to Kafka.
The issue I'm having is that the queue doesn't hold onto messages and simply passes it on (I can see in the relevant statistics that the queue processed the messages) even when the Kafka cluster can't be reached and the action suspends. The librdkafka library - used by omkafka - also has its own queue, but this fills up and message loss occurs. I've also tried to rather set the queue on the ruleset itself, but that too makes no difference.
Below is the queue configuration within the action(type="omkafka" ...) definition.
# Queue Settings
queue.type="LinkedList" # In-memory queue with dynamic memory allocation
queue.spoolDirectory="/var/spool/rsyslog" # Path for disk-assisted queue
queue.size="65000" # Total queue size
queue.highwatermark="60000" # Write messages to disk if highwatermark breached
queue.lowwatermark="50000" # Use in-memory queue again below this mark
queue.filename="kafka_output" # Enables disk-assisted queue and names it
queue.maxdiskspace="18g" # Maximum disk space disk-queue can use
queue.saveonshutdown="on" # Save queue contents to disk on shutdown
Any assistance in this regard would be greatly appreciated.
Kind Regards
---
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.