Have you considered processing the files in parallel using a demand-based algorithm rather than trying to predict their runtime? For example, have X processes, and each time one of the processes finishes, it can simply consume one of the remaining files?

I've had success with it in the past. I generally do it something like:

while (1) {
    # Are we supposed to pause or stop?
    if (-E "/tmp/pause") {
        sleep 10;
        next;
    }
    last if -E "/tmp/STOP";

    # Find a file to process
    my @filist = glob("*.input");

    while (my $file = shift @filist) {
        # try to claim the file
        if (rename $file, "$file.working.$$") {
            # Successful, so go handle it
            process_file("$file.working.$$");
            last;
        }
        # Didn't get the file (someone else might've claimed it just
        # as we tried to), so just loop to the next one...
    }
 }
[download]

The rename is essentially to let the OS handle the task of serializing the processes: on all operating systems I generally use, renaming (on a local filesystem) is considered to be an "atomic" operation, so only one process would "win" the file, and other processes would have to go and try the next one.

The advantages of demand-based sharing are:

• Bad runtime predictions won't cause task starvation
• It balances itself automatically, so it's easy to add or remove processes based on system load.

...roboticus

When your only tool is a hammer, all problems look like your thumb.

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.