Marcello has asked for the wisdom of the Perl Monks concerning the following question:

Hi wise monks,

First, a short description of the perl application. At startup a server process forks of approx. 5 child processes. These persistent child processes (let's name them A, B, C, D and E) each query a MySQL database queue to see if messages are waiting. Simplified database record example: 

Message     Process
abc         A
def         D
ghi         D
jkl         C

[download]
Each child process queries the database approx. 2 times per second, processes only its own messages and deletes the record from the queue, so process A only processes the records where the Process field equals A etc.

Everything is working fine, but the database is queried a lot. Five processes means approx. 5x2 = 10 queries per second just to determine if there are any messages waiting for the process.

A more efficient solution would be to let the server process do the querying, and then inform the corresponding child process(es) that messages are waiting to be processed: 

SELECT DISTINCT(Process) FROM Queue;

[download]
Then a child process only accesses the database when needed, preventing a lot of database selects.

Now my question: I was thinking about using signals. Each time the server process finds messages in the queue, it sends a signal to the corresponding child process. The child process set a flag upon receiving the signal: 

$SIG{USR1} = \&CatchSignal;

...

sub CatchSignal {
    my $signal = shift;

    if ($signal eq "USR1") {
        $SIG_CHECK_DATABASE = 1;
    }
}

...

if ($SIG_CHECK_DATABASE) {
    # Process all messages from the database queue
    $SIG_CHECK_DATABASE = 0;
}

[download]


- Has anybody used a similar approach using signals?
- Is it a good solution?
- What are the disadvantages of such a solution?
- Any other options?

Any information is highly appreciated!

TIA, Marcel

Replies are listed 'Best First'.
Re: Database queueing and signals
by matija (Priest) on Apr 07, 2004 at 08:15 UTC
    I'm vary of using signals too much. When you use signals you must take care about all manners of signal handlers, worry about signals coming whil you're still processing the previous signal etc.

    My preferred approach would be to open a pipe to each process - then the process can just block on reading from the pipe - when a message comes in, it can go off and process it. There is no need to worry that messages arriving too quickly will overwrite/override each other, and since a lot of processes use the "block on reading socket/pipe" model of waiting for data, it is well understood and well optimized for.

[reply]
      Hi,

      I don't need to worry about signals coming while still processing the previous signal. Since the server process checks the database queue twice per second, it does not matter if you "loose" a signal every now and then.

[reply]
Re: Database queueing and signals
by sgifford (Prior) on Apr 07, 2004 at 14:58 UTC

    I don't see any problems with this design. You have to be a little careful what you do in a signal handler, but setting a flag variable should be safe on all systems.

    The question is, what do the child processes do while they're waiting to receive the signal? You don't want them busy-waiting, or they'll tie up the CPU. You can probably make something work with sleep, or select. The pause system call is what you really want, but native Perl doesn't provide an interface to it. You may be able to code one up in XS, or just make do with other functions.

[reply]
[d/l]
[select]
      Thanks for your input.

      Child processes indeed sleep for 200 milliseconds after querying the database, to prevent a high CPU/database load: 
      select(undef, undef, undef, 0.2);

      [download]
[reply]
[d/l]
Re: Database queueing and signals
by gnork (Scribe) on Apr 07, 2004 at 15:58 UTC
    I'd go for the pipe solution.
    Signals can be tricky, although they should be safe in 5.8.3, so you dont have to worry about a signal interrupting signals. This is handled by perl now (please someone correct me if I am wrong).

    Regards
    Jan

    cat /dev/world | perl -e "(/(^.*? \?) 42\!/) && (print $1))"
    errors->(c)
[reply]
Re: Database queueing and signals
by iburrell (Chaplain) on Apr 07, 2004 at 21:34 UTC
    I wouldn't use signals for IPC. There are other IPC options that are easier to use.

    Pipes or sockets between the parent and child processes should work well. Have the clients read from the pipe, process the messages, and wait on reading. The parent would write to the children when it finds a message after polling.

[reply]

Back to Seekers of Perl Wisdom