Ok, here's the breakdown:

The system that this particular script is running on is a SUN StarFire which has 8 x 400MHz SOMBRA CPU Modules and 8GB RAM.

This system is constantly recieving database archive log files from several servers, currently we are recieving from 15 database servers. In peak times, a single server can send as many as 12 x 250MB archive logs per minute. Multiply this value by 15 and we have alot of archive logs on this box. The number of servers sending logs is projected to grow drasticly.

These logs need to be compressed and rsync'd off onto another box offsite (for recovery purposes).

What the scripts main objective is, is to scan a directory structure for archive logs recieved, to compress the archive logs, test the compressed archive log, then rsync it to the remote offsite server.

To run the scan of the directories and compressions in parallel, I figured using queue's would be a good option, as the scanning could be continous, and if anything new is recieved, just to pop it on the queue, then with a thread to process this queue, i could continually compress these logs until the queue is depleted. However, at the rate these archive logs are coming into the system, I need to compress more logs in parallel. Based on the number of CPU's in the system, I was hoping to open 4 simultaneous compressions of the logs, leaving the other 2 available for compression testing (2 threads here) and OS functionality.

I hope this explains it a little better. If you need more info, please dont hesitate to raise it here.

Thanks
Asgaroth

In reply to Re^4: Using Thread::Conveyor and Parallel::ForkManager by Asgaroth
in thread Using Thread::Conveyor and Parallel::ForkManager by Asgaroth

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.