Re^4: Use of do() to run lots of perl scripts

Yes I oversaw the fork part until choroba posted his other benchmark.

Do or even require alone are not fast. Reducing the start up of perl might have a time impact but won't change the RAM consumption.

My bet on the biggest time consumer is the filesystem not the compilation. Precompiling really payed off in the 90s, but now?

So using a RAM-disk could have the best cost benefit ratio.

But we are all speculating here, like others repeated over and over again, the OP should be more explicit

what his problems are (startup, ram, ...)
how frequently this happens
what he benchmarked.

I have my doubts that refactoring 500 scripts is an option and even then...

Precompiling them all into the master process would make them vulnerable to global effects in the BEGIN-phase.

Cheers Rolf
_{(addicted to the Perl Programming Language :)

Wikisyntax for the Monastery}

Comment on Re^4: Use of do() to run lots of perl scripts

Replies are listed 'Best First'.
Re^5: Use of do() to run lots of perl scripts by jcb (Parson) on Mar 04, 2021 at 02:54 UTC
So using a RAM-disk could have the best cost benefit ratio. While our questioner has not told us what operating system this is using, modern systems tend to cache frequently used files in RAM. Linux kernels, in particular, will effectively move frequently-accessed files to a RAM disk, available RAM permitting. This is the "dcache" and Linux has had it for decades. I have my doubts that refactoring 500 scripts is an option The work could be done incrementally. The Pareto principle suggests that 20% of the scripts are probably used 80% of the time, with a long tail into the noise. Our questioner also mentioned that they are not all Perl scripts, so presumably it is already known that the largest contributors to server load are Perl scripts, otherwise the entire question is pointless. Precompiling them all into the master process would make them vulnerable to global effects in the BEGIN-phase. ByteLoader does not work like that. You use B::Bytecode to compile the script in advance, load `ByteLoader` itself in the master process, and use `ByteLoader` in each child to load and run the precompiled script. (Actually, the precompiled script can include `use ByteLoader;` so that `fork or do "script.plc";` will cause `ByteLoader` to be correctly invoked.) Global effects in the BEGIN-phase could be an issue for refactoring the scripts into modules, but addressing those issues would be part of refactoring scripts into modules. :-)	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^5: Use of do() to run lots of perl scripts
by jcb (Parson) on Mar 04, 2021 at 02:54 UTC

So using a RAM-disk could have the best cost benefit ratio.

While our questioner has not told us what operating system this is using, modern systems tend to cache frequently used files in RAM. Linux kernels, in particular, will effectively move frequently-accessed files to a RAM disk, available RAM permitting. This is the "dcache" and Linux has had it for decades.

I have my doubts that refactoring 500 scripts is an option

The work could be done incrementally. The Pareto principle suggests that 20% of the scripts are probably used 80% of the time, with a long tail into the noise. Our questioner also mentioned that they are not all Perl scripts, so presumably it is already known that the largest contributors to server load are Perl scripts, otherwise the entire question is pointless.

Precompiling them all into the master process would make them vulnerable to global effects in the BEGIN-phase.

ByteLoader does not work like that. You use B::Bytecode to compile the script in advance, load ByteLoader itself in the master process, and use ByteLoader in each child to load and run the precompiled script. (Actually, the precompiled script can include use ByteLoader; so that fork or do "script.plc"; will cause ByteLoader to be correctly invoked.)

Global effects in the BEGIN-phase could be an issue for refactoring the scripts into modules, but addressing those issues would be part of refactoring scripts into modules. :-)

[reply]
[d/l]
[select]