Kafka with Avro serialization

mithunbh has asked for the wisdom of the Perl Monks concerning the following question:

I am new to Kafka and am trying to send a message using Avro serialization.

I assumed Kafka::Producer would be the answer, but I believe it doesn't support SSL with certificates.

I then looked at Net::Kafka which does support SSL but now it doesn't support Avro serialization. I thought maybe it is like JSON we build a binary blob and send it off but then how am I using a schema registry?

I am probably asking a lot of newbie questions but the perldoc doesn't clarify what combination would work for my use case. If someone has ideas for me to explore, please do share!

Comment on Kafka with Avro serialization

Replies are listed 'Best First'.
Re: Kafka with Avro serialization by InfiniteSilence (Curate) on Jan 09, 2024 at 23:35 UTC
I've never personally used this stuff. The last message broker I used was a proprietary one at a company. However, a quick search on metacpan reveals the following modules: Kafka::Consumer::Avro Kafka::Producer::Avro Now that I have stated the obvious it gives me the opportunity to tell you that Perl message broker solutions exist. I found one called Mercury that says all you need is Perl installed to run it (though the last release seems to have been a few years back...). The Kafka server appears to be downloadable. Have you tried to install the server locally and try to configure it to use perhaps some generic, locally created SSL certificates? I would then try my solution against the local server to gain familiarity with it prior to using a hosted/cloud-based server when first getting started because it will kick out great error messages and hints that might be obscured by the provider. Celebrate Intellectual Diversity	[reply]
Re^2: Kafka with Avro serialization by mithunbh (Novice) on Jan 11, 2024 at 16:37 UTC
Thank you - yes I am aware of Kafka::Producer. As mentioned Kafka::Producer::Avro & Kafka::Producer doesnt seem to do SSL at least perldoc doesnt tell me how to do so. I have access to fully functional Kafka environment for testing so that is not a problem.	[reply]