Loading one big file is slower than loading 56 different small files?

wdsaeghe has asked for the wisdom of the Perl Monks concerning the following question:

Hello monks,

a collegue of mine stumbled upon something curious: in his script (let's call it main.pl) he has 56 subs he needs.

In scenario 1 he put each sub in a different file, each appropriately named getSynthesedata.pl, transformOrdToCSV.pl.

He then loads all the files using 56 different "do" statements:

do 'getSynthesedata.pl';
#times 56 for each other file
[download]

In Scenario 2, he catted all those files (and nothing more) to one script called all.pl, and only loaded that file:

do 'all.pl';
[download]

Now, loading the 56 different files takes less than one second, loading all.pl takes about 5 seconds. We can't really explain it.

Maybe it has to do with multicore?

Does 'do'-ing a small file has certain benefits?

Comment on Loading one big file is slower than loading 56 different small files? Select or Download Code

Replies are listed 'Best First'.
Re: Loading one big file is slower than loading 56 different small files? by BrowserUk (Patriarch) on Jul 30, 2010 at 10:07 UTC
I cannot reproduce your findings. Below, The first run does 56 simple subs in a single file. The second does those same 56 simple subs each in a separate file. The results pretty much what you'd expect. `c:\test>do-t-main.pl Took 0.008248 seconds c:\test>do-t-main.pl Took 0.101460 seconds` [download] The main thing is that for perl to take 5 whole seconds to load and parse 1 file containing 56 subs is quite extraordinary. They are either humongous subs, or perhaps they are causing some kind of exception handling code to be invoked that slows everything up? Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. RIP an inspiration; A true Folk's Guy	[reply] [d/l]
Re: Loading one big file is slower than loading 56 different small files? by JavaFan (Canon) on Jul 30, 2010 at 09:36 UTC
Does 'do'-ing a small file has certain benefits? Not that I can think of. And there's too much unknown to answer the question why, in your case, the big file is that slow. My question is, why use "do"? It's kind of archaic. Why not use `require` (or `use`?). (Not that I expect using require to speed up the loading for the files).	[reply] [d/l] [select]
Re: Loading one big file is slower than loading 56 different small files? by jethro (Monsignor) on Jul 30, 2010 at 09:54 UTC
I presume you are sure that he didn't do the same file 56 times? Without showing us the code we can't even try to explain. I did a short test with very simple files (6 instead of 56 and my times were 0.046s for the single files and 0.023 for the combined file, as expected. On a double-core machine	[reply]