Global variables and SimpleHTTP request handler

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I an writing a simple web server that receives web requests from an application (Scratch), which I want to process into commands and place in a queue. For the queue I'm trying to use a global array, then have a loop watching that queue to process when able. I can not get the global variables to work.

A cut down version of the code is:

package MyServer;
use warnings;
use strict;

use HTTP::Server::Simple::CGI;
use base qw(HTTP::Server::Simple::CGI);

our @command_queue;
sub handle_request {
    my ($self, $cgi) = @_;
    my $cmd_out;

        # Some processing here which populates $cmd_out
    push(@command_queue, $cmd_out);
    myDebug("Command queue has $#command_queue elements");
}

my $pid = MyServer->new($PORT)->background();
my $quit=0;
while($quit==0) {
        print "Commands in the queue = $#command_queue\n";
        sleep(1);
}
[download]

As each request comes in, the debugger reports the queue increasing in size, so command_queue is being maintained between calls to request_handler. However in the queue monitoring loop at the end, command_queue is constantly reported as empty, -1 in size.

How can I share data between the request_handler and another section of the program?

Comment on Global variables and SimpleHTTP request handler Download Code

Replies are listed 'Best First'.
Re: Global variables and SimpleHTTP request handler by Laurent_R (Canon) on Jan 02, 2017 at 07:30 UTC
You don't say much about the process consuming from the queue. May be it is just so fast that your queue monitoring does not get a chance to see anything in it. Just something else: I would tend to put the subroutine taking items from the queue into the same package, in order to encapsulate the details of the queue management into one single place, which would enable you to use a lexical variable for your queue array.	[reply]
Re^2: Global variables and SimpleHTTP request handler by Anonymous Monk on Jan 03, 2017 at 14:44 UTC
Perhaps I should expand a little on what I'm trying to do. I tried to keep my original post down to the bare details. For those who have not come across it, Scratch is a drag and drop programming language for kids. I'm trying to write an extension to it which can allow the control of Neopixels - programmable LEDs. The Scratch extension sends my custom command to a web server with all the parameters in a URL to a local web server. E.g. the scratch block tries to set pixel 2 with the RGB value 255,0,255 then the requested URL is: `/setneo/2/255/0/255` The hardware itself is being driven by an Arduino which I'm communicating with over a serial cable. Scratch can send the requests faster than the arduino can process them so I need my request handler to place them in a queue which will then be sequentially processed. At the moment the queue processor is just the while loop at the end of the example I give, so it is in the same package. Currently it is just reporting on how big the queue is, but once it can see data from request_handler then it will expand to do something more useful. If a global variable can not be used, then I could look at using a file as a processing queue, but that sounds messy. Is there any other way to share data between two forked processes? A future expansion might be for Scratch to wait until the Arduino has processed a command or accept an input from the Arduino, so eventually I might be looking at a two way communication between the two bits of code.	[reply] [d/l]
Re^3: Global variables and SimpleHTTP request handler by Corion (Patriarch) on Jan 03, 2017 at 14:56 UTC
For communication between processes, see perlipc. Maybe you can use non-blocking communication within a single process by using AnyEvent, but it's likely much easier to write the commands to a file or to simply not return a response from your webserver until the command has been processed, thus pushing the requirement for asynchronicity to Scratch. For some value of "easy", using threads can make it easy to have shared access to a common data structure which is modified concurrently. If your Perl has been compiled with thread support, that may give you a nicer approach. I still recommend a command queue approach using Thread::Queue.	[reply]
Re^4: Global variables and SimpleHTTP request handler by Anonymous Monk on Jan 03, 2017 at 21:39 UTC
Re^3: Global variables and SimpleHTTP request handler by 1nickt (Canon) on Jan 03, 2017 at 15:17 UTC
You can share data between two processes spawned by any forker, or threads, using MCE::Shared. See many nodes here at the monastery by marioroy, author of MCE -- for example this demonstration. Here's an example of some code I use to push to a shared array from multiple workers that are also managed by MCE: `use MCE::Loop; use MCE::Candy; my $shared = [ ]; MCE::Loop::init( max_workers => $CONFIG->max_workers_MCE, chunk_size => 100, gather => MCE::Candy::out_iter_array( $shared ), ); mce_loop { my ( $mce, $chunk_ref, $chunk_id ) = @_; my @items_to_gather; foreach my $item ( @{ $chunk_ref } ) { push @items_to_gather, $item; # your logic here } MCE->gather( $chunk_id, @items_to_gather ); } @list;` [download] Hope this helps! The way forward always starts with a minimal test.	[reply] [d/l]
Re: Global variables and SimpleHTTP request handler by haukex (Archbishop) on Jan 02, 2017 at 19:38 UTC
Hi Anonymous, If you look at the source code of `background`, you'll see that it forks off a process to the background. You can't just use global variables to communicate between two forked processes. While it is in theory possible to communicate between the two (perlipc), I doubt that trying to implement this yourself is the easiest road to success. In fact, I've worked with HTTP::Server::Simple a bit, and I can tell you this: even though it works great for simple stuff, if you find you want to do slightly more complex stuff and you start trying to bend the module to your will, you won't have fun. Instead, switching to a "real" web server ends up being much easier, even if it's just a simple PSGI app in plain Plack. In general, it like sounds what you're trying to implement is a common thing, and so there are lots of possible solutions, many with existing modules and software. But if you could tell us more about the big picture of what you're trying to accomplish, that would help us in suggesting alternate solutions. What kind of requests does Scratch send (JSON? custom format?), what kind of commands do you want to execute, what does "process when able" mean, etc. Regards, -- Hauke D	[reply] [d/l]
Re: Global variables and SimpleHTTP request handler by Anonymous Monk on Jan 01, 2017 at 23:37 UTC
How can I share data between the request_handler and another section of the program? use run instead of background	[reply]
Re^2: Global variables and SimpleHTTP request handler by haukex (Archbishop) on Jan 02, 2017 at 19:28 UTC
Hi, use run instead of background This is incorrect, `run` does not return, and doesn't help in sharing data in the way the OP wants. Regards, -- Hauke D	[reply] [d/l]
Re^3: Global variables and SimpleHTTP request handler by Anonymous Monk on Jan 02, 2017 at 22:42 UTC
This is incorrect, run does not return, and doesn't help in sharing data in the way the OP wants. Why should a server daemon return? Run isn't responsible for sharing, run is reponsible for NOT forking, so the sharing of a global works	[reply]
Re^4: Global variables and SimpleHTTP request handler by haukex (Archbishop) on Jan 03, 2017 at 12:18 UTC