Using Perl files for options versus flatfiles?

AK108 has asked for the wisdom of the Perl Monks concerning the following question:

I'm working (as a developer that's relatively new to the project) on making a bulletin board more efficient in speed and memory. For its configuration files, it uses mostly executeable Perl code. These files don't change too often.

I was wondering about the efficiency of requiring in a configuration file versus opening it, reading it, and closing it. To make things more interesting, the built-in open and close routines aren't used. It uses subroutines that provide file-locking and more instead. Using a database isn't an option, as the board uses flatfiles for everything and it makes no sense to require a database.

Also, if using Perl code as options is better, should I use Data::Dumper or a similar module instead of building up the options (which are just strings and mostly true or false) without it?

P.S. if you recommend a module, it needs to run on the old 5.004-ish versions, and with a pure-Perl implementation.

Edit1: Thought of more stuff to add.

Comment on Using Perl files for options versus flatfiles?

Replies are listed 'Best First'.
Re: Using Perl files for options versus flatfiles? by davido (Cardinal) on Jan 30, 2006 at 06:24 UTC
I think if you find yourself trying to optimize how a configuration file is loaded, you might be barking up the wrong tree for improvements of the code's efficiency. Do you know this to be a bottleneck? Or are you just guessing that it is? If startup time is the problem, maybe converting the script over to mod_perl could be more of a long-term solution. I'm not saying you couldn't shave a few nanoseconds off startup time by tweaking how your config file gets loaded. I'm just saying that nanoseconds are small, and you may find microseconds and milliseconds hiding in other places instead. Dave	[reply]
Re: Using Perl files for options versus flatfiles? by strat (Canon) on Jan 30, 2006 at 10:39 UTC
Before benchmarking and tuning, first try to profile your script, i.e. to find out where it "wastes" most of its time. A good starting point may be the module Devel::Dprof (perl >= 5.003): just run your script with `perl -d:DProf script.pl optionalParameters`, which will produce a file tmon.out. Have a look at this file with the command `dprofpp tmon.out` If this doesn't tell you things you didn't know, maybe Devel::SmallProf might be a finer approach. Well, if you know (not guess) the parts that can be optimized, then try to find different Algorithms (require, open, ...) or Code and benchmark these (e.g. with Benchmark); but take care to benchmark a good representation of the data you work with, and not just some special values that might never come Best regards, perl -e "s>>F>e=>y)\martinF)stronat)=>print,print v8.8.8.32.11.32" Edit: g0n - fixed cpan links	[reply] [d/l] [select]
Re: Using Perl files for options versus flatfiles? by ambrus (Abbot) on Jan 30, 2006 at 08:22 UTC
I think it count much more in terms of efficency if you used fastcgi or modperl or a similar persistent system so that the configuration files don't have to be loaded too often.	[reply]
Re: Using Perl files for options versus flatfiles? by glasswalk3r (Friar) on Jan 30, 2006 at 12:55 UTC
The previous monks already gave good advices about using the Perl profiler and Benchmark, so I won't repeat them. After deciding it the program bottleneck is really the configuration file, then you may want to try using Storable and mod_perl/fastCGI to load only once the configuration file into memory. If your configuration file is not big, I can tell you by experience that using Storable will make it slower than parsing everything (but storing it into memory may help). So you should test it and see what is better. Of course, maybe using such a old Perl version may make this impossible, which I'm not so sure. Despite how difficult is for you to update your Perl version, I think is worth to do it: probably you would gain some more performance using it. Alceu Rodrigues de Freitas Junior --------------------------------- "You have enemies? Good. That means you've stood up for something, sometime in your life." - Sir Winston Churchill	[reply]
Re: Using Perl files for options versus flatfiles? by idle (Friar) on Jan 30, 2006 at 08:30 UTC
You can use Time::HiRes for benchmarking. On 5.005 it work.	[reply]