Mailing List Archive

Issues with omkafka
Hello,



I want to stream the content of a file on kafka. I have setup following
architecture:

File -> imfile -> omkafka -> kafka.



My configuration looks like following:

global (workDirectory="$CUSTOMLOCATION")

module (load="imfile")

module (load="omkafka")

input (type="imfile" File="$PATH_TO_FILE" Tag="" PersistStateInterval="10")

action(type="omkafka" topic="mytopic" broker="$KAFKA_BROKER"
errorFile="$CUSOMLOCATION/error_file" statsFile="$CUSTOMLOCATION/stat_file"
resubmitOnFailure="on" failedMsgFile="$CUSTOMLOCATION/failed_msg"
keepFailedMessages="on" TopicConfParam="request.required.acks=all")
confParam=["socket.timeout.ms=10",
"socket.keepalive.enable=true","max.in.flight=1"])

main_queue(queue.workerthreads="1" queue.dequeueBatchSize="30"
"queue.size=50000" queue.minDequeueBatchSize="10"
queue.minDequeueBatchSize.timeout="100")



I noticed few things with Omkafka module:

1. I don't see errorFile, stat_file, failedMsgFile

2. I had a file with 1,000,000 records. It did not stream all messages
to kafka broker.

3. Omkafka module does not send a single message; it sent a batch of
message to kafka (that is fine with me; but it should be documented
somewhere).

4. There is option to append key in the message. I saw a Jira to make
dynamic keys. However, is it possible to append line_number or message
number as a key.



Also, I am going to stream an important file using omkafka module. I guess I
need to tweak my configuration properly but so far, I have not been able to
stream entire file properly. I expect some duplicates but as long as the
content of messages are sent to kafka broker in orderly manner with/without
duplicates is fine.



Can you please help me fix my configuration ?



Thank you,

Jigar Rathod

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Issues with omkafka [ In reply to ]
El vie., 1 feb. 2019 a las 15:37, Jigar Rathod
(<jrathod@interactivebrokers.com>) escribió:
>
> Hello,
>
>
>
> I want to stream the content of a file on kafka. I have setup following
> architecture:
>
> File -> imfile -> omkafka -> kafka.
>
>
>
> My configuration looks like following:
>
> global (workDirectory="$CUSTOMLOCATION")
>
> module (load="imfile")
>
> module (load="omkafka")
>
> input (type="imfile" File="$PATH_TO_FILE" Tag="" PersistStateInterval="10")
>
> action(type="omkafka" topic="mytopic" broker="$KAFKA_BROKER"
> errorFile="$CUSOMLOCATION/error_file" statsFile="$CUSTOMLOCATION/stat_file"
> resubmitOnFailure="on" failedMsgFile="$CUSTOMLOCATION/failed_msg"
> keepFailedMessages="on" TopicConfParam="request.required.acks=all")
> confParam=["socket.timeout.ms=10",
> "socket.keepalive.enable=true","max.in.flight=1"])

Is that the verbatim config or have you replaced the actual pathes?

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Issues with omkafka [ In reply to ]
Thanks for quick response.

The config specified in the mail is not a verbatim config.
I should have mentioned that I generalized the configuration by specifying variables such as $CUSTOMLOCATION, $PATH_TO_FILE, $KAFKA_BROKER. Actual configuration does not have this variables. It has specific filepath /broker url.

-----Original Message-----
From: rsyslog [mailto:rsyslog-bounces@lists.adiscon.com] On Behalf Of Rainer Gerhards
Sent: Friday, February 01, 2019 11:53 AM
To: rsyslog-users
Subject: Re: [rsyslog] Issues with omkafka

El vie., 1 feb. 2019 a las 15:37, Jigar Rathod
(<jrathod@interactivebrokers.com>) escribió:
>
> Hello,
>
>
>
> I want to stream the content of a file on kafka. I have setup following
> architecture:
>
> File -> imfile -> omkafka -> kafka.
>
>
>
> My configuration looks like following:
>
> global (workDirectory="$CUSTOMLOCATION")
>
> module (load="imfile")
>
> module (load="omkafka")
>
> input (type="imfile" File="$PATH_TO_FILE" Tag="" PersistStateInterval="10")
>
> action(type="omkafka" topic="mytopic" broker="$KAFKA_BROKER"
> errorFile="$CUSOMLOCATION/error_file" statsFile="$CUSTOMLOCATION/stat_file"
> resubmitOnFailure="on" failedMsgFile="$CUSTOMLOCATION/failed_msg"
> keepFailedMessages="on" TopicConfParam="request.required.acks=all")
> confParam=["socket.timeout.ms=10",
> "socket.keepalive.enable=true","max.in.flight=1"])

Is that the verbatim config or have you replaced the actual pathes?

Rainer
_______________________________________________
rsyslog mailing list
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7Cc54539b1595e417d9c8408d68865cda9%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=mYgHDPQA%2BqjmcAHxNatpy%2BDA3iFfolH7s9F2CoslR5o%3D&amp;reserved=0
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7Cc54539b1595e417d9c8408d68865cda9%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=QYqsvrYJRX9079mUBOW4fDK7rSZ0AIRjTZcSQYYrRnc%3D&amp;reserved=0
What's up with rsyslog? Follow https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7Cc54539b1595e417d9c8408d68865cda9%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=tIsuBEGS%2BPM5MR2DHvBh%2BuYghu2q2lUK1ICa3KnV%2BuE%3D&amp;reserved=0
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Issues with omkafka [ In reply to ]
El vie., 1 feb. 2019 a las 18:41, Jigar Rathod
(<jrathod@interactivebrokers.com>) escribió:
>
> Thanks for quick response.
>
> The config specified in the mail is not a verbatim config.
> I should have mentioned that I generalized the configuration by specifying variables such as $CUSTOMLOCATION, $PATH_TO_FILE, $KAFKA_BROKER. Actual configuration does not have this variables. It has specific filepath /broker url.

Which version is it? rsyslogd -v will tell.

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Issues with omkafka [ In reply to ]
It's rsyslogd 8.40.0 with lirdkafka-0.11.5.

-----Original Message-----
From: rsyslog [mailto:rsyslog-bounces@lists.adiscon.com] On Behalf Of Rainer Gerhards
Sent: Friday, February 01, 2019 12:46 PM
To: rsyslog-users
Subject: Re: [rsyslog] Issues with omkafka

El vie., 1 feb. 2019 a las 18:41, Jigar Rathod
(<jrathod@interactivebrokers.com>) escribió:
>
> Thanks for quick response.
>
> The config specified in the mail is not a verbatim config.
> I should have mentioned that I generalized the configuration by specifying variables such as $CUSTOMLOCATION, $PATH_TO_FILE, $KAFKA_BROKER. Actual configuration does not have this variables. It has specific filepath /broker url.

Which version is it? rsyslogd -v will tell.

Rainer
_______________________________________________
rsyslog mailing list
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C8090e542a70e4989affc08d6886d4a2e%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=JeRYcMTtKjORvgFU2mnMfIxoM7p8apoD4O9RkobnGZw%3D&amp;reserved=0
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C8090e542a70e4989affc08d6886d4a2e%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=wney6gQXBirIHB8t4ucfoC%2Bf7rBgRVb062euvOZw%2FAY%3D&amp;reserved=0
What's up with rsyslog? Follow https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C8090e542a70e4989affc08d6886d4a2e%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=GOrYvZydPFLn%2FjuuCkSQs%2F%2F829eEsjYlnrBSCeV4VCc%3D&amp;reserved=0
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Issues with omkafka [ In reply to ]
strange - this means we probably need a debug log showcasing the
problem: https://www.rsyslog.com/doc/v8-stable/troubleshooting/howtodebug.html?highlight=debug%20log

Rainer

El vie., 1 feb. 2019 a las 18:56, Jigar Rathod
(<jrathod@interactivebrokers.com>) escribió:
>
> It's rsyslogd 8.40.0 with lirdkafka-0.11.5.
>
> -----Original Message-----
> From: rsyslog [mailto:rsyslog-bounces@lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Friday, February 01, 2019 12:46 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Issues with omkafka
>
> El vie., 1 feb. 2019 a las 18:41, Jigar Rathod
> (<jrathod@interactivebrokers.com>) escribió:
> >
> > Thanks for quick response.
> >
> > The config specified in the mail is not a verbatim config.
> > I should have mentioned that I generalized the configuration by specifying variables such as $CUSTOMLOCATION, $PATH_TO_FILE, $KAFKA_BROKER. Actual configuration does not have this variables. It has specific filepath /broker url.
>
> Which version is it? rsyslogd -v will tell.
>
> Rainer
> _______________________________________________
> rsyslog mailing list
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C8090e542a70e4989affc08d6886d4a2e%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=JeRYcMTtKjORvgFU2mnMfIxoM7p8apoD4O9RkobnGZw%3D&amp;reserved=0
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C8090e542a70e4989affc08d6886d4a2e%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=wney6gQXBirIHB8t4ucfoC%2Bf7rBgRVb062euvOZw%2FAY%3D&amp;reserved=0
> What's up with rsyslog? Follow https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C8090e542a70e4989affc08d6886d4a2e%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=GOrYvZydPFLn%2FjuuCkSQs%2F%2F829eEsjYlnrBSCeV4VCc%3D&amp;reserved=0
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Issues with omkafka [ In reply to ]
Well, scenario is reproducible. I am running Kafka_2.11-2.0.0 on a single machine. While producer is producing the message; stop the Kafka service.

Expected outcome:
Producer would resend messages that were not sent earlier

Current outcome:
Producer does not resend messages when Kafka broker comes back alive.

I will also send you debug log. It will take me some time moving the logs to external network.

-----Original Message-----
From: rsyslog [mailto:rsyslog-bounces@lists.adiscon.com] On Behalf Of Rainer Gerhards
Sent: Friday, February 01, 2019 1:14 PM
To: rsyslog-users
Subject: Re: [rsyslog] Issues with omkafka

strange - this means we probably need a debug log showcasing the
problem: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rsyslog.com%2Fdoc%2Fv8-stable%2Ftroubleshooting%2Fhowtodebug.html%3Fhighlight%3Ddebug%2520log&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=i6t1HpLpPCOZJDvlmvo28qEE5y%2BUAyq5rvFOKM%2BipIY%3D&amp;reserved=0

Rainer

El vie., 1 feb. 2019 a las 18:56, Jigar Rathod
(<jrathod@interactivebrokers.com>) escribió:
>
> It's rsyslogd 8.40.0 with lirdkafka-0.11.5.
>
> -----Original Message-----
> From: rsyslog [mailto:rsyslog-bounces@lists.adiscon.com] On Behalf Of
> Rainer Gerhards
> Sent: Friday, February 01, 2019 12:46 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Issues with omkafka
>
> El vie., 1 feb. 2019 a las 18:41, Jigar Rathod
> (<jrathod@interactivebrokers.com>) escribió:
> >
> > Thanks for quick response.
> >
> > The config specified in the mail is not a verbatim config.
> > I should have mentioned that I generalized the configuration by specifying variables such as $CUSTOMLOCATION, $PATH_TO_FILE, $KAFKA_BROKER. Actual configuration does not have this variables. It has specific filepath /broker url.
>
> Which version is it? rsyslogd -v will tell.
>
> Rainer
> _______________________________________________
> rsyslog mailing list
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.
> adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Cjrathod%
> 40interactivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef
> 837d48e69ba869d84f65a110%7C0&amp;sdata=KgOqRgKsnlHZ9JnMzk4MejGoNgv1V%2
> FneKqOglAKCArk%3D&amp;reserved=0
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rs
> yslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Cjrathod%40inte
> ractivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48
> e69ba869d84f65a110%7C0&amp;sdata=OpV9xI%2BCgMI59RGaFG3tLWFbQKPsX2X%2BD
> bPJGlQzc0I%3D&amp;reserved=0 What's up with rsyslog? Follow
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitt
> er.com%2Frgerhards&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com
> %7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48e69ba869d84f65a110
> %7C0&amp;sdata=Y6rYDRwuYzMkFVO6%2F%2BbS2jUbVkjpXBlwxcieriJT9w0%3D&amp;
> reserved=0 NOTE WELL: This is a PUBLIC mailing list, posts are
> ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.
> adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Cjrathod%
> 40interactivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef
> 837d48e69ba869d84f65a110%7C0&amp;sdata=KgOqRgKsnlHZ9JnMzk4MejGoNgv1V%2
> FneKqOglAKCArk%3D&amp;reserved=0
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rs
> yslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Cjrathod%40inte
> ractivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48
> e69ba869d84f65a110%7C0&amp;sdata=OpV9xI%2BCgMI59RGaFG3tLWFbQKPsX2X%2BD
> bPJGlQzc0I%3D&amp;reserved=0 What's up with rsyslog? Follow
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitt
> er.com%2Frgerhards&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com
> %7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48e69ba869d84f65a110
> %7C0&amp;sdata=Y6rYDRwuYzMkFVO6%2F%2BbS2jUbVkjpXBlwxcieriJT9w0%3D&amp;
> reserved=0 NOTE WELL: This is a PUBLIC mailing list, posts are
> ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=KgOqRgKsnlHZ9JnMzk4MejGoNgv1V%2FneKqOglAKCArk%3D&amp;reserved=0
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=OpV9xI%2BCgMI59RGaFG3tLWFbQKPsX2X%2BDbPJGlQzc0I%3D&amp;reserved=0
What's up with rsyslog? Follow https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=Y6rYDRwuYzMkFVO6%2F%2BbS2jUbVkjpXBlwxcieriJT9w0%3D&amp;reserved=0
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Issues with omkafka [ In reply to ]
Prerequisite:
rsyslog is reading a file that has 1 million lines. For simplicity, each line can be a line number itself. Kafka broker is up & running. Keep queue size small i.e. 50,000.

Scenario:
Run rsyslog for few minutes. It will start sending some messages. Suddenly stop kafka broker.

Observation:
Content from Imfile-state will be updating continuously. Curr_offs will keep increasing. When curr_offs stops increasing; start kafka broker. You will notice that no messages would be published to kafka.

My theory:
Imfile and omkafka are not in sync. Omkafka prints success message whenever it sends a message successfully. Omkafka is aware of what's going on. However, imfile is not aware of anything. It will continue to write to queue continuously. This may be the reason why curr_offs will keep increasing. When it stops increasing; this means imfile has done its job. No more data to read; so it will not put anything in the queue.

I assume if the queue size was huge then this would not have happened. But, producer cannot rely upon that.

Is there a way to tell imfile to stop reading the file; as soon as omkafka detects that kafka broker is not working ?

-----Original Message-----
From: rsyslog [mailto:rsyslog-bounces@lists.adiscon.com] On Behalf Of Jigar Rathod
Sent: Friday, February 01, 2019 1:52 PM
To: 'rsyslog-users'
Subject: Re: [rsyslog] Issues with omkafka

Well, scenario is reproducible. I am running Kafka_2.11-2.0.0 on a single machine. While producer is producing the message; stop the Kafka service.

Expected outcome:
Producer would resend messages that were not sent earlier

Current outcome:
Producer does not resend messages when Kafka broker comes back alive.

I will also send you debug log. It will take me some time moving the logs to external network.

-----Original Message-----
From: rsyslog [mailto:rsyslog-bounces@lists.adiscon.com] On Behalf Of Rainer Gerhards
Sent: Friday, February 01, 2019 1:14 PM
To: rsyslog-users
Subject: Re: [rsyslog] Issues with omkafka

strange - this means we probably need a debug log showcasing the
problem: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rsyslog.com%2Fdoc%2Fv8-stable%2Ftroubleshooting%2Fhowtodebug.html%3Fhighlight%3Ddebug%2520log&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C56b3ed2e9e4f4332015e08d688767216%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=jdBYOXyAZkm4RCEuUp0ouBaMURHySxiJdrMLXeyQhGE%3D&amp;reserved=0

Rainer

El vie., 1 feb. 2019 a las 18:56, Jigar Rathod
(<jrathod@interactivebrokers.com>) escribió:
>
> It's rsyslogd 8.40.0 with lirdkafka-0.11.5.
>
> -----Original Message-----
> From: rsyslog [mailto:rsyslog-bounces@lists.adiscon.com] On Behalf Of
> Rainer Gerhards
> Sent: Friday, February 01, 2019 12:46 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Issues with omkafka
>
> El vie., 1 feb. 2019 a las 18:41, Jigar Rathod
> (<jrathod@interactivebrokers.com>) escribió:
> >
> > Thanks for quick response.
> >
> > The config specified in the mail is not a verbatim config.
> > I should have mentioned that I generalized the configuration by specifying variables such as $CUSTOMLOCATION, $PATH_TO_FILE, $KAFKA_BROKER. Actual configuration does not have this variables. It has specific filepath /broker url.
>
> Which version is it? rsyslogd -v will tell.
>
> Rainer
> _______________________________________________
> rsyslog mailing list
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.
> adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Cjrathod%
> 40interactivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef
> 837d48e69ba869d84f65a110%7C0&amp;sdata=KgOqRgKsnlHZ9JnMzk4MejGoNgv1V%2
> FneKqOglAKCArk%3D&amp;reserved=0
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rs
> yslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Cjrathod%40inte
> ractivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48
> e69ba869d84f65a110%7C0&amp;sdata=OpV9xI%2BCgMI59RGaFG3tLWFbQKPsX2X%2BD
> bPJGlQzc0I%3D&amp;reserved=0 What's up with rsyslog? Follow
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitt
> er.com%2Frgerhards&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com
> %7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48e69ba869d84f65a110
> %7C0&amp;sdata=Y6rYDRwuYzMkFVO6%2F%2BbS2jUbVkjpXBlwxcieriJT9w0%3D&amp;
> reserved=0 NOTE WELL: This is a PUBLIC mailing list, posts are
> ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.
> adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Cjrathod%
> 40interactivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef
> 837d48e69ba869d84f65a110%7C0&amp;sdata=KgOqRgKsnlHZ9JnMzk4MejGoNgv1V%2
> FneKqOglAKCArk%3D&amp;reserved=0
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rs
> yslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Cjrathod%40inte
> ractivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48
> e69ba869d84f65a110%7C0&amp;sdata=OpV9xI%2BCgMI59RGaFG3tLWFbQKPsX2X%2BD
> bPJGlQzc0I%3D&amp;reserved=0 What's up with rsyslog? Follow
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitt
> er.com%2Frgerhards&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com
> %7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48e69ba869d84f65a110
> %7C0&amp;sdata=Y6rYDRwuYzMkFVO6%2F%2BbS2jUbVkjpXBlwxcieriJT9w0%3D&amp;
> reserved=0 NOTE WELL: This is a PUBLIC mailing list, posts are
> ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C56b3ed2e9e4f4332015e08d688767216%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=bu%2Fe1CDg%2FO5dBmEa2Bb27KTRQvXfPGVC4MjTEVGESJ0%3D&amp;reserved=0
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C56b3ed2e9e4f4332015e08d688767216%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=VmgZR9BE6qW2gKjs8F1geAMx63DxjUVOKzhyGrzVcAc%3D&amp;reserved=0
What's up with rsyslog? Follow https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C56b3ed2e9e4f4332015e08d688767216%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=gQHgqVdYx%2BNK9dEkUq3G7gUFMV6z1cswgFf3Re6bIts%3D&amp;reserved=0
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C56b3ed2e9e4f4332015e08d688767216%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=bu%2Fe1CDg%2FO5dBmEa2Bb27KTRQvXfPGVC4MjTEVGESJ0%3D&amp;reserved=0
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C56b3ed2e9e4f4332015e08d688767216%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=VmgZR9BE6qW2gKjs8F1geAMx63DxjUVOKzhyGrzVcAc%3D&amp;reserved=0
What's up with rsyslog? Follow https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C56b3ed2e9e4f4332015e08d688767216%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=gQHgqVdYx%2BNK9dEkUq3G7gUFMV6z1cswgFf3Re6bIts%3D&amp;reserved=0
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Issues with omkafka [ In reply to ]
Should I create an issue for omkafka on github?

-----Original Message-----
From: rsyslog [mailto:rsyslog-bounces@lists.adiscon.com] On Behalf Of Jigar Rathod
Sent: Friday, February 01, 2019 3:17 PM
To: 'rsyslog-users'
Subject: Re: [rsyslog] Issues with omkafka


Prerequisite:
rsyslog is reading a file that has 1 million lines. For simplicity, each line can be a line number itself. Kafka broker is up & running. Keep queue size small i.e. 50,000.

Scenario:
Run rsyslog for few minutes. It will start sending some messages. Suddenly stop kafka broker.

Observation:
Content from Imfile-state will be updating continuously. Curr_offs will keep increasing. When curr_offs stops increasing; start kafka broker. You will notice that no messages would be published to kafka.

My theory:
Imfile and omkafka are not in sync. Omkafka prints success message whenever it sends a message successfully. Omkafka is aware of what's going on. However, imfile is not aware of anything. It will continue to write to queue continuously. This may be the reason why curr_offs will keep increasing. When it stops increasing; this means imfile has done its job. No more data to read; so it will not put anything in the queue.

I assume if the queue size was huge then this would not have happened. But, producer cannot rely upon that.

Is there a way to tell imfile to stop reading the file; as soon as omkafka detects that kafka broker is not working ?

-----Original Message-----
From: rsyslog [mailto:rsyslog-bounces@lists.adiscon.com] On Behalf Of Jigar Rathod
Sent: Friday, February 01, 2019 1:52 PM
To: 'rsyslog-users'
Subject: Re: [rsyslog] Issues with omkafka

Well, scenario is reproducible. I am running Kafka_2.11-2.0.0 on a single machine. While producer is producing the message; stop the Kafka service.

Expected outcome:
Producer would resend messages that were not sent earlier

Current outcome:
Producer does not resend messages when Kafka broker comes back alive.

I will also send you debug log. It will take me some time moving the logs to external network.

-----Original Message-----
From: rsyslog [mailto:rsyslog-bounces@lists.adiscon.com] On Behalf Of Rainer Gerhards
Sent: Friday, February 01, 2019 1:14 PM
To: rsyslog-users
Subject: Re: [rsyslog] Issues with omkafka

strange - this means we probably need a debug log showcasing the
problem: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rsyslog.com%2Fdoc%2Fv8-stable%2Ftroubleshooting%2Fhowtodebug.html%3Fhighlight%3Ddebug%2520log&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=ljVCMZaXvTUrCwPg3GGmFQvmHvSjSzyKQVjS60i816I%3D&amp;reserved=0

Rainer

El vie., 1 feb. 2019 a las 18:56, Jigar Rathod
(<jrathod@interactivebrokers.com>) escribió:
>
> It's rsyslogd 8.40.0 with lirdkafka-0.11.5.
>
> -----Original Message-----
> From: rsyslog [mailto:rsyslog-bounces@lists.adiscon.com] On Behalf Of
> Rainer Gerhards
> Sent: Friday, February 01, 2019 12:46 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Issues with omkafka
>
> El vie., 1 feb. 2019 a las 18:41, Jigar Rathod
> (<jrathod@interactivebrokers.com>) escribió:
> >
> > Thanks for quick response.
> >
> > The config specified in the mail is not a verbatim config.
> > I should have mentioned that I generalized the configuration by specifying variables such as $CUSTOMLOCATION, $PATH_TO_FILE, $KAFKA_BROKER. Actual configuration does not have this variables. It has specific filepath /broker url.
>
> Which version is it? rsyslogd -v will tell.
>
> Rainer
> _______________________________________________
> rsyslog mailing list
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.
> adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Cjrathod%
> 40interactivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef
> 837d48e69ba869d84f65a110%7C0&amp;sdata=KgOqRgKsnlHZ9JnMzk4MejGoNgv1V%2
> FneKqOglAKCArk%3D&amp;reserved=0
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rs
> yslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Cjrathod%40inte
> ractivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48
> e69ba869d84f65a110%7C0&amp;sdata=OpV9xI%2BCgMI59RGaFG3tLWFbQKPsX2X%2BD
> bPJGlQzc0I%3D&amp;reserved=0 What's up with rsyslog? Follow
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitt
> er.com%2Frgerhards&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com
> %7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48e69ba869d84f65a110
> %7C0&amp;sdata=Y6rYDRwuYzMkFVO6%2F%2BbS2jUbVkjpXBlwxcieriJT9w0%3D&amp;
> reserved=0 NOTE WELL: This is a PUBLIC mailing list, posts are
> ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.
> adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Cjrathod%
> 40interactivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef
> 837d48e69ba869d84f65a110%7C0&amp;sdata=KgOqRgKsnlHZ9JnMzk4MejGoNgv1V%2
> FneKqOglAKCArk%3D&amp;reserved=0
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rs
> yslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Cjrathod%40inte
> ractivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48
> e69ba869d84f65a110%7C0&amp;sdata=OpV9xI%2BCgMI59RGaFG3tLWFbQKPsX2X%2BD
> bPJGlQzc0I%3D&amp;reserved=0 What's up with rsyslog? Follow
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitt
> er.com%2Frgerhards&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com
> %7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48e69ba869d84f65a110
> %7C0&amp;sdata=Y6rYDRwuYzMkFVO6%2F%2BbS2jUbVkjpXBlwxcieriJT9w0%3D&amp;
> reserved=0 NOTE WELL: This is a PUBLIC mailing list, posts are
> ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=%2BjvppADq2BKIsbwT8u4jBJmfIT8VljXmjpdwKfhJR2E%3D&amp;reserved=0
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=p8HTfDNBWJcftbR2Z8gCbLv62dSRA1uGEZF76UQgzWs%3D&amp;reserved=0
What's up with rsyslog? Follow https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=88A3sy1dT7K9BjtcOczzgfAWE5puTZLWqJd%2FkeF1tMc%3D&amp;reserved=0
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=%2BjvppADq2BKIsbwT8u4jBJmfIT8VljXmjpdwKfhJR2E%3D&amp;reserved=0
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=p8HTfDNBWJcftbR2Z8gCbLv62dSRA1uGEZF76UQgzWs%3D&amp;reserved=0
What's up with rsyslog? Follow https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=88A3sy1dT7K9BjtcOczzgfAWE5puTZLWqJd%2FkeF1tMc%3D&amp;reserved=0
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=%2BjvppADq2BKIsbwT8u4jBJmfIT8VljXmjpdwKfhJR2E%3D&amp;reserved=0
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=p8HTfDNBWJcftbR2Z8gCbLv62dSRA1uGEZF76UQgzWs%3D&amp;reserved=0
What's up with rsyslog? Follow https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&amp;sdata=88A3sy1dT7K9BjtcOczzgfAWE5puTZLWqJd%2FkeF1tMc%3D&amp;reserved=0
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.