Let's suppose you are always forking and the RAM consumption of all those forks (which means cloning the run-time engine, which needs 1.5MB++ on my system) is no issue.

Let's suppose further it's the start-up time that matters.

Like already explained does do FILE still imply overhead for

but do is just a glorified eval `cat FILE` mechanism.

So your MASTER process could just keep all those scripts and used modules° in a big hash $file{SCRIPT}

Now after forking to child THIS_SCRIPT_1 you only need to eval this script directly to compile it and you have deprived yourself already of point A the FS overhead.

(As a further optimization you could now empty the hash %file to release memory of the child, tho I'm not sure if this would pay of)

And now - provided that this script is started many times - you can fork again from THIS_SCRIPT_1 after compilation and THIS_SCRIPT_2 is executed with a clean start-up context and terminated at the end.

Every time MASTER needs this particular script to be run again, he needs to communicate to child1 which forks again to a grandchild which is run again with precompiled code, solving the overhead of point B.

Now it's up to you to decide if you want to keep all these 500 child-forks of level 1 constantly alive by reserving 1-2GB of RAM for it, I'd rather make it depended of a frequency count of the MASTER. (And I still doubt that compile time is an issue nowadays, but YMMV)

I still think you are most probably reinventing the wheel here, because such strategies have certainly already be discussed in the context of web-servers.

But it could solve your issues by buying time with space. and you will still need to benchmark all of this.

Cheers Rolf
(addicted to the Perl Programming Language :)
Wikisyntax for the Monastery

update

°) used modules B2 is a bit more complicated, anything with hooking into @INC could be used. You said these scripts are "simple", do they always use the same modules? In that case require the common ones in MASTER, this will do the compilation (hopefully without global side-effects)

see also App::FatPacker et al...

In reply to Re: Use of do() to run lots of perl scripts by LanX
in thread Use of do() to run lots of perl scripts by chrestomanci

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.