Mailing List Archive

omkafka and custom data serialization
Hi All,

I'm using rsyslog with omkafka to push logs into a Kafka topic in json
format, which works well.

Is there a way to serialize the logs using a custom (non-textual)
serialization format before pushing into Kafka. (protobuf is what I'm after)

Otherwise, what is a good approach to achieve the same.

Cheers,
Sam
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: omkafka and custom data serialization [ In reply to ]
rsyslog has a very powerful engine to let you format your data, but I don't know
that it can do protobuf (it's text only), it would not be that hard to create a
module that would format a message as protobuf (and Adiscon does professional
services development at reasonable costs if you are interested in sponsoring
such a module)

David Lang

On Thu, 24 Jun 2021, Sam S via rsyslog wrote:

> Is there a way to serialize the logs using a custom (non-textual)
> serialization format before pushing into Kafka. (protobuf is what I'm after)
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: omkafka and custom data serialization [ In reply to ]
Will familiarise with the module system, and get back with questions.
This is a side project at the moment, but I'll keep the tip in mind.
Cheers,
Sam


On Fri, Jun 25, 2021 at 1:04 AM David Lang <david@lang.hm> wrote:

> rsyslog has a very powerful engine to let you format your data, but I
> don't know
> that it can do protobuf (it's text only), it would not be that hard to
> create a
> module that would format a message as protobuf (and Adiscon does
> professional
> services development at reasonable costs if you are interested in
> sponsoring
> such a module)
>
> David Lang
>
> On Thu, 24 Jun 2021, Sam S via rsyslog wrote:
>
> > Is there a way to serialize the logs using a custom (non-textual)
> > serialization format before pushing into Kafka. (protobuf is what I'm
> after)
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.