Experience with the Kafka interface modules

Wiggins has asked for the wisdom of the Perl Monks concerning the following question:

Folks I really hate to do this, but I need advice on the 'Kafka.pm' modules (interface to Apache Kafka Messaging). They do not install because of multiple failed tests.

I an working on RHEL7 (new OS for me), and have all the needed support packages installed. But the Kafka module is not installed due to numerous failed tests!

There is a mysterious statement in the README:

The following environment variable changes the way Kafka Benchmark tes
+t behave:
 
   # The root directory of the installation Kafka 0.8+
   KAFKA_BASE_DIR
[download]

Nothing more. Is that just an empty directory? is this the only inference that 'kafka' is expected to be installed on all system where this Perl module is installed?
Well, without this set, final testing fails because that ENV is not set. Why not just say that the ENV is required? I pointed that ENV to '/opt/kafkaEmptyDir', and made it to test 07_cluster.t which failed, as did all the following tests

There is no Apache Kafka installed on this client system, the Kafka is installed on a separate Server system.

---Update------------

I have things working, and I thought that if you called up this thread, you might like a tip on testing that your code really works. I installed Kafka with a 'walk-through' blog entry, so I am not at all familiar with Kafka itself. But, in order to get things talking, the Kafka configuration file "server.properties" needs to have the 2 'hostname' related items:

host.name=kafka
advertised.host.name=kafka
[download]

And then I set /etc/hosts (on both machines) so that 'kafka' resolved to the IP of the Kafka server. Set it to some name resolvable by either the DNS or the /etc/hosts file on both the remote Kafka and the Perl client machine.

Good luck

It is always better to have seen your target for yourself, rather than depend upon someone else's description.

Comment on Experience with the Kafka interface modules Select or Download Code

Replies are listed 'Best First'.
Re: Experience with the Kafka interface modules by Corion (Patriarch) on May 08, 2016 at 17:44 UTC
You don't show which part of the docs you're looking at. Kafka contains the README, which documents the following environment variables for the benchmark test: `TESTING The following environment variables change the way Kafka Benchmark tes +t behave ("t/99_usage/03_kafka_bench.t"): # the IO timeout in secs KAFKA_BENCHMARK_TIMEOUT = 0.5 # minimum length of test messages KAFKA_BENCHMARK_LEN_MIN = 200 # maximum length of the test messages KAFKA_BENCHMARK_LEN_MAX = 200 # number of messages for batch testing KAFKA_BENCHMARK_PACKAGE = 5000 # number of messages for a single test KAFKA_BENCHMARK_SINGLE = 5` [download] I find them fairly self-descriptive and I wouldn't worry about a benchmark test failing too much.	[reply] [d/l]
Re^2: Experience with the Kafka interface modules by Wiggins (Hermit) on May 08, 2016 at 20:17 UTC
We are looking at 2 different files. Mine is the result of a 'module' load using 'cpan install Kafka'; while you are referencing a 'Distribition' Kaffka-0.12.tar. There is a Kafka::Mock module that has the same -0.12 nomenclature. The CPAN command "i /Kafka/ " shows both. In 'root/.cpan/build/Kafka-09010-xxxxxx'there is only one README with: `... TESTING The following environment variable changes the way Kafka Benchmark tes +t behave: # The root directory of the installation Kafka 0.8+ KAFKA_BASE_DIR SIMPLE BENCHMARK ...` [download] Which is is the most current or correct one, or what are the content differences, I don't know... ++++ UPDATE+++++ I just tried installing the Distribution, and it installed without comment. ++++ UPDATE-2 +++++ The distribution did not contain ALL of the modules. Kafka::Message.pm and Kafka::Connection, at least, were not installed. I believe other utility pieces were also missing. I did a 'cpan install Kafka::Message' and numerous other modules were also installed. The sample program in the Kafka::Producer man page does pass a syntax check, but always ends with the error `Error: (-1004) Can't bind: topic = 'ort2'` [download] A tcpdump shows a good TCP connection with 50+ packets exchanged. And I tried pre-defined topics and unique new topics More study needed... It is always better to have seen your target for yourself, rather than depend upon someone else's description.	[reply] [d/l] [select]