in reply to Apache or Daemon in Perl ?

I have the following image of your setup:

   collector (mod_perl) ->   MySQL   ->  sender (LWP?)

[download]

If your clients support HTTP/1.1, it might be worthwhile to use the LWP::ConnCache in your sender, so you don't have to reopen the connection to your client(s) on every new message. I don't know if Apache has a built-in method for connecting/sending via sockets - if it has, using that mechanism or fudging it into LWP::ConnCache might speed up the process a lot. You maybe should make the sender into different processes that (all) poll the database for new messages as soon as they've sent the current batch of messages - this will generate a lot of load on the database, but you can easily decouple the reception and sending of the messages that way.

Other than that, I don't really see much of a better way, as you have already decoupled your system and use a versatile scheduling mechanism, the database... You could try to separate the system further by putting the database and sender onto two different machines, or better on a 4 CPU machine, so you have three CPUs, enough RAM and enough network capability to keep all parts saturated.

Replies are listed 'Best First'.
Re^2: Apache or Daemon in Perl ?
by Solo (Deacon) on Jan 18, 2005 at 16:34 UTC
    Is that 100 requests/sec sustained or burst?

    I have to agree with Corion's design here.

    collector (mod_perl) -> MySQL -> sender (LWP?)

    Use a mod_perl handler with persistent database connections (ala Apache::DBI) to write the incoming info to a suitably fast database (like MySQL). Have a separate daemon written in Perl (or something compiled if performance dictates) that reads from the DB and sends the responses.

    Make sure the DB stores timestamps for request received and response transmitted. Your sending daemon fetches a 'worklist' by querying for records w/o transmitted timestamp. The table can be archived/summarized/cubed periodically during slow periods. If there are no slow periods, you might need to investigate 'breaking-the-mirror' type solutions to get an offline snapshot for reporting.

    --Solo

    --
    You said you wanted to be around when I made a mistake; well, this could be it, sweetheart.
[reply]
Re^2: Apache or Daemon in Perl ?
by szabgab (Priest) on Jan 18, 2005 at 13:45 UTC
    The sender is some in-house written thing using IO::Socket and I think I cannot keep the sockets open to these devices as they might need that for other things. But that might be a good direction for improvement.
[reply]

In Section Seekers of Perl Wisdom