Hello,
I want to stream the content of a file on kafka. I have setup following
architecture:
File -> imfile -> omkafka -> kafka.
My configuration looks like following:
global (workDirectory="$CUSTOMLOCATION")
module (load="imfile")
module (load="omkafka")
input (type="imfile" File="$PATH_TO_FILE" Tag="" PersistStateInterval="10")
action(type="omkafka" topic="mytopic" broker="$KAFKA_BROKER"
errorFile="$CUSOMLOCATION/error_file" statsFile="$CUSTOMLOCATION/stat_file"
resubmitOnFailure="on" failedMsgFile="$CUSTOMLOCATION/failed_msg"
keepFailedMessages="on" TopicConfParam="request.required.acks=all")
confParam=["socket.timeout.ms=10",
"socket.keepalive.enable=true","max.in.flight=1"])
main_queue(queue.workerthreads="1" queue.dequeueBatchSize="30"
"queue.size=50000" queue.minDequeueBatchSize="10"
queue.minDequeueBatchSize.timeout="100")
I noticed few things with Omkafka module:
1. I don't see errorFile, stat_file, failedMsgFile
2. I had a file with 1,000,000 records. It did not stream all messages
to kafka broker.
3. Omkafka module does not send a single message; it sent a batch of
message to kafka (that is fine with me; but it should be documented
somewhere).
4. There is option to append key in the message. I saw a Jira to make
dynamic keys. However, is it possible to append line_number or message
number as a key.
Also, I am going to stream an important file using omkafka module. I guess I
need to tweak my configuration properly but so far, I have not been able to
stream entire file properly. I expect some duplicates but as long as the
content of messages are sent to kafka broker in orderly manner with/without
duplicates is fine.
Can you please help me fix my configuration ?
Thank you,
Jigar Rathod
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
I want to stream the content of a file on kafka. I have setup following
architecture:
File -> imfile -> omkafka -> kafka.
My configuration looks like following:
global (workDirectory="$CUSTOMLOCATION")
module (load="imfile")
module (load="omkafka")
input (type="imfile" File="$PATH_TO_FILE" Tag="" PersistStateInterval="10")
action(type="omkafka" topic="mytopic" broker="$KAFKA_BROKER"
errorFile="$CUSOMLOCATION/error_file" statsFile="$CUSTOMLOCATION/stat_file"
resubmitOnFailure="on" failedMsgFile="$CUSTOMLOCATION/failed_msg"
keepFailedMessages="on" TopicConfParam="request.required.acks=all")
confParam=["socket.timeout.ms=10",
"socket.keepalive.enable=true","max.in.flight=1"])
main_queue(queue.workerthreads="1" queue.dequeueBatchSize="30"
"queue.size=50000" queue.minDequeueBatchSize="10"
queue.minDequeueBatchSize.timeout="100")
I noticed few things with Omkafka module:
1. I don't see errorFile, stat_file, failedMsgFile
2. I had a file with 1,000,000 records. It did not stream all messages
to kafka broker.
3. Omkafka module does not send a single message; it sent a batch of
message to kafka (that is fine with me; but it should be documented
somewhere).
4. There is option to append key in the message. I saw a Jira to make
dynamic keys. However, is it possible to append line_number or message
number as a key.
Also, I am going to stream an important file using omkafka module. I guess I
need to tweak my configuration properly but so far, I have not been able to
stream entire file properly. I expect some duplicates but as long as the
content of messages are sent to kafka broker in orderly manner with/without
duplicates is fine.
Can you please help me fix my configuration ?
Thank you,
Jigar Rathod
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.