Marcello has asked for the wisdom of the Perl Monks concerning the following question:

Hi monks,

I'm trying to develop a fast queue for my clients in Perl. Clients can put requests in this queue, and a Perl program will check the queue constantly and process the requests.

Currently, I use a file-based queue:

Advantages:
- Processing speed

Disadvantage:
- Accepting speed (100-200 files/sec)
- Flexibility

I would like to improve the accepting speed by using a database and DBI, since INSERT statements are fast. The only drawback is that the processing speed of the requests will be around 10-20 requests per second, since I testing it on a simple database with SELECT statements.

What is the best approach to setup a queue with maximum accepting speed like INSERTS and maximum processing speed like in using files?

Hope I made my problem clear.
TIA

Best regards,
Marcel

Replies are listed 'Best First'.
Re: Fast queue logic
by dws (Chancellor) on Dec 06, 2002 at 10:20 UTC
    What is the best approach to setup a queue with maximum accepting speed like INSERTS and maximum processing speed like in using files?

    The fastest way is to write a queue management service which keeps the queue entries in memory. Clients access the service via socket. That makes polling quick, but isn't particularly fault-tolerant. You need to persist the queue at some point. I'd start off with files, and see if that's good enough.

[reply]
Re: Fast queue logic
by Zaxo (Archbishop) on Dec 06, 2002 at 10:40 UTC

    You need to flesh out your requirements a bit. You list 100-200 'files/sec', bur I don't see where those files come from. Are they requested by name, or is their content part of the request? Are these all new connections, or just a few busy clients? What protocol is this?

    That is a pretty rapid request rate. You will need to handle several requests per timeslice on linux, and that is only considering CPU usage. Typically, I/O rate is the limiting factor in client/server applications.

    How much hardware can you throw at this? How much do you have now? Your data rate can be met by dedicated apache servers, is that what you want?

    After Compline,
    Zaxo

[reply]
      Time for some more explanation:

      Clients connect using TCP. The request from clients are now stored in files for processing, that is why I said 100-200 files/sec. This is currently the fasted rate at which request can be stored. This I would like to improve by using a database with INSERTS, which is considerably faster. But to process these queued requests (either in files or database), the filesystem is much faster again. So I need to find the best of both worlds, actually.

      If you need more information, I will be happy to give you it.

      Regards,
      Marcel
[reply]
        Maybe I'm missing something too, but how are these "requests" arriving and what subsequenct processing happens, ie how are the results returned, etc?

        Somthing like SOAP::Lite connected to HTTP::Daemon or Apache with some kind of tied hash would seem to be the obvious choice, but without a full description of the entire processing cycle it's kind of difficult to say "here's the definitive answer...".

        rdfield

[reply]
Re: Fast queue logic
by perrin (Chancellor) on Dec 06, 2002 at 15:51 UTC
    You think that moving from flat files to a relational database will make adding things faster? Flat files are nearly always faster than a database for this kind of simple operation. If files seem to be going too slowly for you, you should profile your code to see if there's something taking up the time. There are also a number of nodes on this site that talk about speeding up file access.
[reply]
Re: Fast queue logic
by PodMaster (Abbot) on Dec 06, 2002 at 10:18 UTC
    use DB_File or BerkeleyDB. Good day.

    update: it's always a good idea to look on CPAN.

    MJD says you can't just make shit up and expect the computer to know what you mean, retardo!
    ** The Third rule of perl club is a statement of fact: pod is sexy.

[reply]
Re: Fast queue logic
by rcaputo (Chaplain) on Dec 10, 2002 at 17:37 UTC

    You can't beat a named pipe (FIFO) for speed, but there are two requirements:

    1. Your queue items must be relatively small (usually less than 512 bytes).
    2. Your queue reader and writer processes must all be on the same machine.

    -- Rocco Caputo - troc@pobox.com - poe.perl.org

[reply]

Back to Seekers of Perl Wisdom