AUTOLOAD for variables?

dpuu has asked for the wisdom of the Perl Monks concerning the following question:

Is it possible to do the equivalent of an AUTOLOAD, but for accesses to undefined (un-existing) variables, not functions?

The basic scenario is that I'm maintaining some code that uses Perl as the format of a configuration file, and some people's config files access variables without pre-declaring them. There's a performance problem with the overlying script, becasue its taking a long time to calculate/fetch the values of all the variables that could be accessed in this config file, even though only a small number of variables are used in a given config file. We'd like to avoid modifying user-environments, and fix the performance by tweaking the master-script.

The current code that reads a config-file simply does:

  sub read_config_file {
    my ($filename) = @_;
    package ConfigFile;
    do "$filename";
    if (length $@) { ... }
  }
[download]

So what I'd really like to do is to trap accesses to non-existing (scalar) variables in package "ConfigFile", and substitute a value that I determine using some expensive function call. E.g.

  tie %ConfigFile::, "ValueFetcher"
[download]

Unfortunately, I can't work out how to tie a symbol table (nor even if I really want to)

One option I've considered is to simply tie every known variable that anyone might want to access, and thus defer fetching the actual value until later. But this feels very, um, un-lazy. (And even working out what variables could be accessed can be somewhat expensive).

If there's no better way to do it, then I guess I'll have to bite the bullet and find an intrusive solution; and then get everyone to change their config files. But doing that with hundreds of users is not something I look forward to.

--Dave
Opinions my own; statements of fact may be in error.

Comment on AUTOLOAD for variables? Select or Download Code

Replies are listed 'Best First'.
Re: AUTOLOAD for variables? by tachyon (Chancellor) on Jul 08, 2004 at 03:22 UTC
When you access a variable that does not exist perl will AUTOVIVIFY it (ie create it for you). Although TheDamian was doing some work on an AUTOVIVIFY sub (equivalent to AUTOLOAD but for vars) AFAIK you can't currently catch access to undefined vars. It sounds like you have some significant design issues. Perl code for config files + lots of users is an interesting (scary) concept. What is to stop someone inserting arbitrary code that will then run with the perms of you master script? Your explanation of the problem you are trying to solve may make sense to you but is far from clear at this end. You have a performance problem and you say it is loading the config files. Are you sure? What is actually in these files? Why do you think delaying loading will speed things up? Is this a CGI? Are you using mod_perl? ????? cheers tachyon	[reply]
Re^2: AUTOLOAD for variables? by dpuu (Chaplain) on Jul 08, 2004 at 04:19 UTC
I was trying not to get into too many details -- they aren't terribly important -- except for finding alternative solutions that don't involve the AUTOVIVIFY method. Yes, the loads of users of perl-config files is kinda scary -- especially when you see how some people take advantage of having perl. The context has nothing to do with web. It's actually the test environment for our hardware (chip) development. People configure the test scripts by grabing configuration from a database of chip-specific information -- and that database has grown somewhat big over the years (along with the number of scripts that access it). You are right that there are some serious design issues -- the sort that take a fair amount of time to sort out. We basically have to trawl through a deeply tested directory structure, over nfs, to find about 50K files in the leaves -- then parse those files into a flat namespace for the config file (its not a real database -- which may be part of the problem. Its not quite as bad as it sounds, because we can reduce the scanning with some caching) Then a config file uses only a small number of the values we supply. Multiply this onto a farm with multiple thousand CPUs, and soon you're talking real money! It really is tremendously ugly (and it may be impossible to reverse-engineer a spec). I was hoping to find a quick fix to use as a stop-gap measure while sorting out the real problems. Perhaps if there is no quick fix, then there's a greater incentive to apply the resources needed to clean it up. I guess this is what happens when a startup becomes big, quickly. --Dave Opinions my own; statements of fact may be in error, or may be deliberately obscured.	[reply]
Re^3: AUTOLOAD for variables? by tachyon (Chancellor) on Jul 08, 2004 at 04:58 UTC
flat namespace for the config file (its not a real database -- which may be part of the problem It really depends on how it actually works in practice but it sounds like your end data structure could work better. Depending on all sorts of factors an RDBMS or using a flatfile that stores a hash data structure, serialised with say Storable may help. With the Storable suggestion you would parse the data into a hash (very fast access) then store it into a flatfile with Storable. Anything that needs access just loads the file and gets the whole config hash ready to go. The net result is you effectively spend memory to gain speed. cheers tachyon	[reply]
Re: AUTOLOAD for variables? by Chmrr (Vicar) on Jul 08, 2004 at 04:48 UTC
It's possible to do, if you hook into the warning that gets tossed when a variable is only used once. For example: `#!/usr/bin/perl -w use warnings; BEGIN { use Lingua::EN::Words2Nums; $SIG{__WARN__} = sub { if ($_[0] =~ /Name "(.)" used only once/) { my $var = $1; my ($num) = $var =~ /::(\S+)/; $num =~ tr/_/ /; $$var = words2nums($num); } else { warn @_; } }; } print $five + $seventeen, "\n";` [download] However, this is firmly in the "sick trick" arena, rather than "maintable code." Heed the other monks' advice to look for better ways to go about this. Update:* Oh, yeah -- this obviously only works if you only refer to the variable once in your code. This may be more trouble than it is worth, obviously. It also won't catch things like `$foo = "five"; print $$foo;`	[reply] [d/l] [select]
Re^2: AUTOLOAD for variables? by dpuu (Chaplain) on Jul 08, 2004 at 06:00 UTC
I like the trick but, unfortunately, the evil of my users far surpasses it. Its not uncommon to see code such as: `my $value= ${ uc join "_", @$reg_spec{ qw( device reg field ) } };` [download] You really don't want to see some of the more nasty code. Honest. Especially not the 70000 line regex (its one saving grace is the /x modifier). And perhaps not the script that parses another script's --verbose output. Or perhaps ... --Dave Opinions my own; statements of fact may be in error.	[reply] [d/l]
Re^3: AUTOLOAD for variables? by dragonchild (Archbishop) on Jul 08, 2004 at 13:22 UTC
Create an API. Create a set of expected standards. Give your users 3-6 months to comply with those standards, then issue the smack-down. Period. You can't do everything for everyone. ------ We are the carpenters and bricklayers of the Information Age. Then there are Damian modules.... sigh* ... that's not about being less-lazy -- that's about being on some really good drugs -- you know, there is no spoon.* - flyingmoose I shouldn't have to say this, but any code, unless otherwise stated, is untested	[reply]