Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

Interprocess messaging with Net::Clacks

by cavac (Priest)
on Oct 04, 2018 at 11:52 UTC ( #1223514=CUFP: print w/replies, xml ) Need Help??

So, your project is going fine, your codebase is groing fast. But now your have the problem that some of your processes have to communicate with each other. Maybe, some temperature sensor needs to report its sensor value every few seconds to the central heating system. Maybe the central heating system needs to know if the windows are open and close them before heating the house. Another process wants to count how many times the door has been opened and log the sum once a minute...

Net::Clacks to the rescue!

The Net::Clacks modules implement a client/server based interprocess messaging. Without going too much into the internals of the protocol, a client can either send notifications ("event xy has just happened") or values ("the reading for sensor xy is now 42"). Other clients may (or may not) choose to listen to those broadcasts.

Net::Clacks also implements Memcached-like value storage and retrieval. So instead of broadcasting, a value can be stored, read out, incremented, decremented and deleted.

A note on security: Currently, the system only implements a shared-secret type thing (all clients in a clacks network use the same username/password). This will get changed in the future. I'm planning to make it so that you can override the authentication checks with your own function and return which permissions the client has. But that is not yet implemented.

Let's do a simple example project: Server, chatclient, chatbot and a clock process to trigger some actions at the start of every minute.

First of all, we need a server. For this, we need an XML config file and a bit of Perl code.

#!/usr/bin/env perl #---AUTOPRAGMASTART--- use 5.020; use strict; use warnings; use diagnostics; use mro 'c3'; use English; use Carp; our $VERSION = 4.1; use Fatal qw( close ); use Array::Contains; #---AUTOPRAGMAEND--- my $isDebugging = 0; if(defined($ARGV[1]) && $ARGV[1] eq "--debug") { $isDebugging = 1; } use Net::Clacks::Server; my $configfile = shift @ARGV; croak("No Config file parameter") if(!defined($configfile) || $configf +ile eq ''); my $worker = Net::Clacks::Server->new($isDebugging, $configfile); $worker->init; $worker->run;

And the config (saved as clacks_master.xml, we'll add a slaveserver later on):

<clacks> <appname>Clacks Master</appname> <ip>127.0.0.1</ip> <port>18888</port> <pingtimeout>600</pingtimeout> <interclackspingtimeout>10</interclackspingtimeout> <ssl> <cert>exampleserver.crt</cert> <key>exampleserver.key</key> </ssl> <username>exampleuser</username> <password>unsafepassword</password> <throttle> <maxsleep>5000</maxsleep> <step>1</step> </throttle> <hosts> <developmentbox> <ip>128.0.0.1</ip> <ip>::1</ip> </developmentbox> </hosts> </clacks>

Oh, and yes, we need an OpenSSL certificate as well. For this demo we will make a self signed one. Not very secure, but sufficient for a proof of concept:

openssl req -new -newkey rsa:4096 -x509 -sha256 -days 36500 -nodes -ou +t exampleserver.crt -keyout exampleserver.key

Now let's start the server:

perl server.pl clacks_master.xml

First client will be the "clock.pl" process, this is basically a "write only" thing, it only sends a notify() to all clients listening to it at the start of every minute.

#!/usr/bin/env perl #---AUTOPRAGMASTART--- use 5.020; use strict; use warnings; use diagnostics; use mro 'c3'; use English; use Carp; our $VERSION = 4.1; use Fatal qw( close ); use Array::Contains; #---AUTOPRAGMAEND--- use Net::Clacks::Client; use Term::ReadKey; use Time::HiRes qw(sleep); use Data::Dumper; my $username = 'exampleuser'; my $password = 'unsafepassword'; my $applicationname = 'clock'; my $is_caching = 0; my $chat = Net::Clacks::Client->new('127.0.0.1', 18888, $username, $pa +ssword, $applicationname, $is_caching); my $clockname = 'example::notify'; my $last = ''; while(1) { my $now = getCurrentMinute(); if($now ne $last) { $chat->notify($clockname); $chat->ping(); $last = $now; } $chat->doNetwork(); while((my $msg = $chat->getNext())) { if($msg->{type} eq 'disconnect') { print '+++ Disconnected by server, reason given: ', $msg-> +{data}, "\n"; last; } } sleep(0.2); } sub getCurrentMinute { my ($sec,$min, $hour, $mday,$mon, $year, $wday,$yday, $isdst) = lo +caltime time; $year += 1900; $mon += 1; $mon = doFPad($mon, 2); $mday = doFPad($mday, 2); $hour = doFPad($hour, 2); $min = doFPad($min, 2); return "$year-$mon-$mday $hour:$min"; } sub doFPad { my ($val, $len) = @_; while(length($val) < $len) { $val = '0' . $val; } return $val; }

Notice, we also have to send the occasional ping(). This tells the server we are still alive, so it doesn't close the connection. You usually do this in your main loop. I personally had a bug in which the client kept sending some data while it was stuck in a subroutine, so this is some sort of insurance that the clients higher brain functions still get called...

Another thing to note is the doNetwork() call. Message sending and receiving over the network doesn't happen automatically, messages get buffered in the client library. This reduces network traffic when handling lots of small messages by bundling them into bigger packets.

Now, let's do the actual chat client. This is quite similar, except we listen() to some messages at the beginning and also handle the extra message types. Oh, and of course, take line-by-line user input and send those chat messages to the server.

#!/usr/bin/env perl #---AUTOPRAGMASTART--- use 5.020; use strict; use warnings; use diagnostics; use mro 'c3'; use English; use Carp; our $VERSION = 4.1; use Fatal qw( close ); use Array::Contains; #---AUTOPRAGMAEND--- use Net::Clacks::Client; use Term::ReadKey; use Time::HiRes qw(sleep); use Data::Dumper; my $username = 'exampleuser'; my $password = 'unsafepassword'; my $applicationname = 'chatclient'; my $is_caching = 0; my $chat = Net::Clacks::Client->new('127.0.0.1', 18888, $username, $pa +ssword, $applicationname, $is_caching); #print 'Connected to server. Info given: ', $chat->getServerinfo(), "\ +n"; my $chatname = 'example::chat'; my $clockname = 'example::notify'; $chat->listen($chatname); $chat->listen($clockname); $chat->ping(); $chat->doNetwork(); my $nextping = time + 60; while(1) { my $line = ReadLine -1; if(defined($line)) { chomp $line; if(length($line)) { last if(uc $line eq 'QUIT' || uc $line eq 'EXIT'); $chat->set($chatname, $line); } } if($nextping < time) { $chat->ping(); $nextping = time + 60; } $chat->doNetwork(); while((my $msg = $chat->getNext())) { if($msg->{type} eq 'set' && $msg->{name} eq $chatname) { print '>>> ', $msg->{data}, "\n"; } elsif($msg->{type} eq 'notify' && $msg->{name} eq $clockname +) { print "*** Another minute has passed ***\n"; } elsif($msg->{type} eq 'disconnect') { print '+++ Disconnected by server, reason given: ', $msg-> +{data}, "\n"; last; } } sleep(0.2); }

At the beginning, we listen() to the two message names. On user input, we send it as a chat message to everyone else listening to the chat. Roughly every 60 seconds we ping() the server (keepalive again). Then we send and receive with doNetwork() and check to see if we got any interesting messages.

We could simplify the checks quite a bit since there are only two well defined messages (one 'notify', one 'set'), but i implemented the full checking here to illustrate the principle.

Last but not least, let's implement the most brain-dead chatbot we can think of. It repeats messages containing "simon" with "Simon says". It also implements a counter in Clacks for counting chat messages. Every time it gets a clock notification, it sends the number of chat messages seen since the last notification as a chat message and decreases the counter accordingly. (I could just set it to zero, but using decrement with the previously read out value is the correct thing to do, especially in cases where the counter is increased by some other program).

#!/usr/bin/env perl #---AUTOPRAGMASTART--- use 5.020; use strict; use warnings; use diagnostics; use mro 'c3'; use English; use Carp; our $VERSION = 4.1; use Fatal qw( close ); use Array::Contains; #---AUTOPRAGMAEND--- use Net::Clacks::Client; use Term::ReadKey; use Time::HiRes qw(sleep); use Data::Dumper; my $username = 'exampleuser'; my $password = 'unsafepassword'; my $applicationname = 'chatbot'; my $is_caching = 0; my $chat = Net::Clacks::Client->new('127.0.0.1', 18888, $username, $pa +ssword, $applicationname, $is_caching); #print 'Connected to server. Info given: ', $chat->getServerinfo(), "\ +n"; my $chatname = 'example::chat'; my $clockname = 'example::notify'; my $countname = 'chatbot::linecount'; $chat->listen($chatname); $chat->listen($clockname); $chat->ping(); $chat->doNetwork(); my $nextping = time + 60; while(1) { my $line = ReadLine -1; if(defined($line)) { chomp $line; if(length($line)) { last if(uc $line eq 'QUIT' || uc $line eq 'EXIT'); $chat->set($chatname, $line); } } if($nextping < time) { $chat->ping(); $nextping = time + 60; } $chat->doNetwork(); while((my $msg = $chat->getNext())) { if($msg->{type} eq 'set' && $msg->{name} eq $chatname) { # Increment count for every chat message $chat->increment($countname); # very non-AI answer implementation of "simon says" if($msg->{data} =~ /simon/i) { $chat->set($chatname, 'RoboSimon says: ' . $msg->{data +}); $chat->increment($countname); } } elsif($msg->{type} eq 'notify' && $msg->{name} eq $clockname +) { # Every minute, retrieve our current count, send a chat me +ssage and decrease line count accordingly my $linecount = $chat->retrieve($countname); if(!defined($linecount)) { $linecount = 0; $chat->store($countname, 0); } $chat->set($chatname, $linecount . ' new chat messages dur +ing the last minute'); $chat->decrement($countname, $linecount); } elsif($msg->{type} eq 'disconnect') { print '+++ Disconnected by server, reason given: ', $msg-> +{data}, "\n"; last; } } $chat->doNetwork(); sleep(1); }

Ok, sure, this chat system will never get much users, because it just doesn't have good features. It's just a demo of some of the Net::Clacks capabilities.

But "what if" you actually get more users or processes that a single service or server can handle? No problem here, either. Net::Clacks::Server has what is called an "Interclacks" mode, which can be used to run in master/slave mode. When a slave starts up and connects (or reconnects after a network error) to the master, it drops it's locally stored variables and resyncs from the master.

After that, every message or stored variable you set/send/notify gets automatically synced to the master. A slave can also work as a master node at the same time, so you are able to chain nodes in a tree like pattern if you really so desire.

So, lets make a slave node. In our case, it will run on the same host with a different port (but you can trivially change it to run on a different server). All we need is an additional XML config file:

<clacks> <appname>Clacks Master</appname> <ip>127.0.0.1</ip> <port>18889</port> <pingtimeout>600</pingtimeout> <interclackspingtimeout>10</interclackspingtimeout> <ssl> <cert>exampleserver.crt</cert> <key>exampleserver.key</key> </ssl> <username>exampleuser</username> <password>unsafepassword</password> <throttle> <maxsleep>5000</maxsleep> <step>1</step> </throttle> <master> <ip>127.0.0.1</ip> <port>18888</port> </master> </clacks>

And then start it:

perl server.pl clacks_slaveserver.xml

So far, i have had Net::Clacks in use on a couple of live systems for over 2 years, most of them running 1000+ clacks connections at the same time. At least one of them having a constant 24/7 message rate of roughly 5000 messages per minute. Pretty much no crashes in the last few months. Obviously, you need to test, test, test before you decide to use it for your own systems.

"For me, programming in Perl is like my cooking. The result may not always taste nice, but it's quick, painless and it get's food on the table."

Replies are listed 'Best First'.
Re: Interprocess messaging with Net::Clacks
by zentara (Archbishop) on Oct 04, 2018 at 15:20 UTC
    Wow, this could be very useful in artificial intelleigence and automation systems.

    I'm not really a human, but I play one on earth. ..... an animated JAPH

      I'd say so. My first test/demo was my virtual coffee factory over on my blog.

      The browsers uses Websockets to communicate with the webserver. And the webserver uses Clacks to communicate with the single instance of the virtual factory. It sync across all webbrowsers that show the page. If you put it in manual mode, you can control it yourself, but it's intentionally hard. But with a friend or two helping you from their computers sharing the tasks, it's quite possible to run the thing in manual mode.

      I'm also running a clacks-over-the-internet experiment where you can watch Henry the pear tree grow up. Currently it's not always on because my ISP works on upgrading local cabling and my bandwidth is very limited most of these days. Also, i might need to clean the camera again.

      "For me, programming in Perl is like my cooking. The result may not always taste nice, but it's quick, painless and it get's food on the table."
Re: Interprocess messaging with Net::Clacks
by soonix (Canon) on Oct 04, 2018 at 16:59 UTC

      Yes, of course.

      The protocol (which is still missing proper documentation in the tarball) even implements an OVERHEAD method with compatible flags to the book. So sending

      OVERHEAD GNU Terry Pratchett

      actually works.

      "For me, programming in Perl is like my cooking. The result may not always taste nice, but it's quick, painless and it get's food on the table."
Re: Interprocess messaging with Net::Clacks
by stevieb (Canon) on Oct 04, 2018 at 21:58 UTC

    Given I have very high interest in automation systems (zentara mentioned automation/AI), this presentation appears promising.

    There are a couple of things I'd like to add to one of my distributions for inter-proc communications, so I'll play around a bit with this.

    Nice job.

    -stevieb

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: CUFP [id://1223514]
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (4)
As of 2022-01-20 17:24 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    In 2022, my preferred method to securely store passwords is:












    Results (57 votes). Check out past polls.

    Notices?