chrestomanci has asked for the wisdom of the Perl Monks concerning the following question:

Greetings wise brothers.

I have an existing perl forking server that listens on a TCP/IP socket, forks a new process for each incoming connection and that process spends some time (5-60 minutes) handing the remote client.

I would like to convert the server to a systemd socket activated service. This is so that I can upgrade the code for the server via a Debian package upgrade without interrupting the child workers and their ongoing transaction.

My current version is configured to run as a long running systemd daemon. I have prepared a minimal version that shows how the current service works.

#!/perl # Minimal forking server - Shows what is needed in a systemd socket ve +rsion of the ET daemon. use strict; use warnings; use English; use IO::Socket::INET; use Socket qw(SOMAXCONN SOL_SOCKET SO_KEEPALIVE); use POSIX qw(:sys_wait_h ECHILD EINTR); # Flags used by the signal handlers my $quit = 0; # Signal handling functions sub REAPER { my $child; while ((my $waitedpid = waitpid(-1,WNOHANG)) > 0) { my $message = "reaped $waitedpid" . ($CHILD_ERROR ? " with exi +t $CHILD_ERROR" : ""); print "$message\n"; } } $SIG{CHLD} = \&REAPER; # loathe sysV $SIG{TERM} = sub { $quit = 1 }; sub handle_connection { my ($sock) = @_; setsockopt($sock, SOL_SOCKET, SO_KEEPALIVE, 1); ######################################################## # # Important. We need to know the remote IP address. # ######################################################## my $remote_addr = inet_ntoa ($sock->peeraddr()); printf $sock "Greetings %s what is your command?\n", $remote_addr; # Code here to check that $remote_addr is authorised to connect, a +nd which remote commands it may call. # Need to close the socket & refuse futher commands $remote_addr i +s not authorised. while( my $cmd = <$sock> ) { if( $cmd =~ m/^sleep (\d+)/ ) { my $snooze_time = $1; printf $sock "Will sleep for $snooze_time seconds\n"; sleep $snooze_time; print $sock "... ZZZ ... Yawn ... What next oh master?\n"; } # NB: In real life there are lots of other commands and many t +ake some time to process. else { print $sock "I did not understand that. What next oh maste +r?\n"; } } # Socket connection closed. exit(0); # This is in a forked child. } # MAIN # Create the server socket to listen for incoming connections my $s = IO::Socket::INET->new( LocalPort => 1234, Listen => SOMAXCONN, ReuseAddr => 1, Timeout => 5, # seconds Blocking => 0, # After the timeout the accept() call will r +eturn regardless ); $s->listen(); print "ET daemon startup - Listening on port 1234 - My PID:$PID\n"; ACCEPT: while ( 0 == $quit ) { my $new_s = $s->accept(); if( defined $new_s ) { print "Accepted socket connection on port 1234\n"; redo ACCEPT if $! == ECHILD or $! == EINTR; # fork off to handle the new connection my $pid = fork(); if ($pid == 0) { print "CHILD: Calling handle_connection()\n"; handle_connection($new_s); } else { print "PARENT: Forked off child PID:$pid\n"; close($new_s); print "PARENT: Closed the socket & waiting for new connect +ions.\n"; } } else { # The 5 second timeout on the the socket accept() call has exp +ired, or a HUP signal was received. } } print "Caught TERM will quit now\n";

I believe it is possible to configure systemd to handle the socket and the fork for each incoming connection on behalf of my service, so I would just need the handle_connection() method and the necessary boilerplate to retrieve the remote IP address and close the connection if the remote IP address is not authorised to connect.

I found some example code by one of the systemd developers, but it is all in C and makes use of systemd specific C libraries and data structures, so I am having trouble understanding it. Can anyone point me to a perl or python example that is easier to follow?

Replies are listed 'Best First'.
Re: Systemd socket activated service in perl
by tybalt89 (Monsignor) on Jan 19, 2024 at 22:39 UTC

    Check out Systemd::Daemon.

    However since I am familiar with xinetd and not systemd, I did your server in xinetd instead.

    #!/usr/bin/perl use strict; # https://perlmonks.org/?node_id=11157093 use warnings; $| = 1; # NOTE # slightly reformatted from your code - $sock not needed while( my $cmd = <> ) { if( $cmd =~ m/^sleep (\d+)/ ) { my $snooze_time = $1; printf "Will sleep for $snooze_time seconds\n"; sleep $snooze_time; print "... ZZZ ... Yawn ... What next oh master?\n"; } # NB: In real life there are lots of other commands and many take some + time to process. else { print "I did not understand that. What next oh master?\n"; } } __END__ # # unlisted service # (goes in /etc/xinetd.d) NOTE change user, server, and only_from # if this file changed, use 'systemctl reload xinetd' so prev still ru +n # using 'restart' will kill running servers, 'reload' will not # service unlisted-pm11157093 { type = UNLISTED socket_type = stream protocol = tcp wait = no user = rick log_type = FILE /dev/null server = /home/rick/code/pm11157093.pl port = 1234 disable = no only_from = 127.0.0.1 192.168.1.0/24 }

    Note that xinetd does listening and forking for you, runs your child with the data socket connected to STDIN and STDOUT, and even has an 'only_from' parameter to verify remote machines.
    Leaves very little to do in your server :)

    UPDATE: corrected spelling of 'xinitd' to 'xinetd'

      And after some digging, here's a systemd version. It uses the very same server code in /home/rick/code/pm11157093.pl from this node's parent (change to your liking), and is called pfs (for perl forking server).

      Make file /etc/systemd/system/pfs.socket

      # perl forking server https://perlmonks.org/?node_id=11157093 [Unit] Description=perl forking socket [Socket] ListenStream=1234 Accept=yes

      and file /etc/systemd/system/pfs@.service

      # perl forking server https://perlmonks.org/?node_id=11157093 [Unit] Description=perl forking server Requires=pfs.socket [Service] ExecStart=-/home/rick/code/pm11157093.pl StandardInput=socket

      run systemctl start pfs.socket and fire up a telnet to test it.

      Some extra config lines may be required to start automatically at boot, plus systemctl enable pfs.socket

      Perhaps a systemd expert can fill in any holes I've missed.

      UPDATE: the remote addr is in $ENV{REMOTE_ADDR} (see man systemd.socket)

      ++ for using the right tool for the job! When all you have is the systemd hammer, everything starts looking like a nail. We never used to be so hamstrung.


      🦛

      Thanks for the link to Systemd::Daemon I will check it out.

      Sadly, the example code you have given me for xinetd won't be sufficient. I really need the remote IP address as for the production service the same request will get substantially different answers depending on who is asking, and that answer can change from minute to minute via a database query. A simple whitelist of allowed IP addresses is no where near sufficient.

        The remote host is in $ENV{REMOTE_HOST}

Re: Systemd socket activated service in perl
by NERDVANA (Priest) on Jan 19, 2024 at 17:50 UTC
    The article you linked is an example where systemd passes the listen socket to the service. The service would still need to accept connections and fork off children to handle them. Are you sure that systemd offers a mode where it accepts and launches a new service each time a connections is received?

    If your main goal is to let long-lived processes keep running on an old version while a new version is deployed and new connections use the new code, all you really need to do is execute a new perl process instead of forking. Or in other words, fork like you do above, then exec($^X, $script_name, @args); and put sub handle_connection in that script file. You can replace the file at any time and new connections get the new implementation without interrupting current ones.


    So... I was curious and asked ChatGPT. It thinks your solution could be this simple:

    If you're looking for a setup similar to inetd, where a new process is spawned for each incoming connection, systemd has a feature called socket-activated services with per-connection instances. This is achieved by using the Accept=yes option in the socket unit file. Here's a simplified example: Create a socket unit file, let's say example-socket.socket:
    [Unit] Description=Example Socket [Socket] ListenStream=12345 Accept=yes [Install] WantedBy=sockets.target
    This sets up a socket to listen on port 12345 and spawns a new service for each incoming connection. Create a service unit file, for example example-service@.service:
    [Unit] Description=Example Service [Service] ExecStart=/path/to/your/executable StandardInput=socket
    Note the %i in the service file, which represents the instance identifier. systemd will replace this with a unique identifier for each connection. Enable and start the socket:
    systemctl enable example-socket.socket systemctl start example-socket.socket
    Now, systemd will spawn a new service instance for each incoming connection on the specified socket.
    If that is correct (which clearly it forgot to use a '%i' in the service file... and I don't see any linkage between the socket unit and the service unit...) then the top-level of your script could be reduced to just
    handle_conenction(\*STDIN);

      The linkage is the name of the units, in this case 'example-socket'. It can be overridden, but it is the default.

      UPDATE: note that a wrong file name is given by chatGPT - 'example-server@.service' should be 'example-socket@.service'

Re: Systemd socket activated service in perl
by chrestomanci (Priest) on Jan 21, 2024 at 10:50 UTC

    Thanks to everyone for the help here, especially tybalt89.

    Bringing it all together and testing it on an Ubuntu 22.04 VM, I have this simplified perl script:

    #!/usr/bin/perl # Minimal systemd socket activated service # Put together based on responses from: https://www.perlmonks.org/?nod +e_id=11157093 use strict; use warnings; use English; $OUTPUT_AUTOFLUSH = 1; # AKA $| # Important. We need to know the remote IP address. my $remote_addr = $ENV{REMOTE_ADDR}; printf "Greetings %s what is your command?\n", $remote_addr; # printf "%s = %s\n", $_, $ENV{$_} foreach sort keys %ENV; # Us +ed to get the rest of the environment. # Code here to check that $remote_addr is authorised to connect, and w +hich remote commands it may call. # Need to close the socket & refuse futher commands $remote_addr is no +t authorised. while( my $cmd = <> ) { if( $cmd =~ m/^sleep (\d+)/ ) { my $snooze_time = $1; printf "Will sleep for $snooze_time seconds\n"; sleep $snooze_time; print "... ZZZ ... Yawn ... What next oh master?\n"; } # NB: In real life there are lots of other commands and many take +some time to process. else { print "I did not understand that. What next oh master?\n"; } }

    With two systemd unit files:

    The socket:

    /etc/systemd/system/pfs.socket

    [Unit] Description=perl forking socket PartOf=pfs.service [Socket] ListenStream=2345 BindIPv6Only=both Accept=yes MaxConnections=12 [Install] WantedBy=sockets.target

    And the service:

    /etc/systemd/system/pfs@.service

    [Unit] Description=perl forking server After=network.target nss-user-lookup.target Requires=pfs.socket [Service] ExecStart=/root/path/to/minimal_systemd_socket_activated_service.pl StandardInput=socket [Install] Also=pfs.socket WantedBy=multi-user.target

    Once all three files are installed on the sever:

    • Make sure the perl script is executable and has a valid hashbang line
    • systemctl daemon-reload
    • systemctl enable pfs.service
    • systemctl enable pfs.socket
    • systemctl start pfs.socket

    It should then be possible to test the service:

    echo "sleep 30" | nc localhost 2345 -NC

    Notes and observations:

    Like a CGI script, when the socket activated service is running, STDIN and STDOUT are the socket to the remote client, so printf debugging and logging does not work as expected.

    Similar to a cron job, the socket activated service gets very few environment variables set. It is possible to add Environment="variable=value" entries into the Service stanza of the pfs@.service file.

    The service file should have an @ symbol in the name. It is not a mistake. In other types of systemd units, the @ symbol means that the service is parameterised, and we should expect to see a %i placeholder in the service unit file. For the socket activated service, the socket sets up the parameters when it invokes the service.

    I connected to the service to start a lengthy sleep, then ran systemctl status pfs.socket

    This gave an extra Triggers entry that is not show when the socket is idle.

    root@Ubuntu:~# systemctl status pfs.socket
    ● pfs.socket - perl forking socket
         Loaded: loaded (/etc/systemd/system/pfs.socket; enabled; vendor preset: enabled)
         Active: active (listening) since Sun 2024-01-21 10:18:43 GMT; 9min ago
       Triggers: ● pfs@5-127.0.0.1:2345-127.0.0.1:39394.service
         Listen: :::2345 (Stream)
       Accepted: 6; Connected: 1;
          Tasks: 0 (limit: 9361)
         Memory: 8.0K
            CPU: 1ms
         CGroup: /system.slice/pfs.socket
    

    I was then able to get the status of the temporary service:

    root@Ubuntu:~# systemctl status pfs@5-127.0.0.1:2345-127.0.0.1:39394.service
    ● pfs@5-127.0.0.1:2345-127.0.0.1:39394.service - perl forking server (127.0.0.1:39394)
         Loaded: loaded (/etc/systemd/system/pfs@.service; disabled; vendor preset: enabled)
         Active: active (running) since Sun 2024-01-21 10:27:42 GMT; 11s ago
    TriggeredBy: ● pfs.socket
       Main PID: 3225 (minimal_systemd)
          Tasks: 1 (limit: 9361)
         Memory: 1.2M
            CPU: 5ms
         CGroup: /system.slice/system-pfs.slice/pfs@5-127.0.0.1:2345-127.0.0.1:39394.service
                 └─3225 /usr/bin/perl /root/path/to/minimal_systemd_socket_activated_service.pl
    

    And investigate the environment variables of that service

    root@Ubuntu:~# cat /proc/3225/environ | xargs -0 -n1
    LANG=en_GB.UTF-8
    LANGUAGE=en_GB:en
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
    REMOTE_ADDR=127.0.0.1
    REMOTE_PORT=39394
    INVOCATION_ID=73c1fa9b1e8b45c0a557538dce370fd6
    SYSTEMD_EXEC_PID=3225