jonc has asked for the wisdom of the Perl Monks concerning the following question:

Hello again and thanks for the continued support,

I am building a CGI script to go online, and it is slow, so I used a profiler to find how to make it better. The problem is, I don't know what I could do...

As you'll see, CGI::AUTOLOAD takes the most time (over half)... What does this mean? Is it something I can control?

Secondly is the File::Find. I tried readdir, but it produced similar results. In the works is eliminating the need to loop through files (by making one large file), I hope this will make it more efficient.

Here is part of the output (I doubt you guys need the whole thing) from one test by Devel::Profile :

time elapsed (wall): 27.9298 time running program: 27.9106 (99.93%) time profiling (est.): 0.0192 (0.07%) number of calls: 4170 %Time Sec. #calls sec/call F name 73.95 20.6410 7 2.948712 CGI::AUTOLOAD 25.48 7.1128 3626 0.001962 <anon>:...Perl/5.10.0/File/Fin +d.pm:717 0.27 0.0746 1 0.074612 File::Find::_find_dir 0.08 0.0216 1 0.021615 <anon>:...bles/mytestsearch201 +1.cgi:25 0.05 0.0142 7 0.002028 CGI::_compile 0.05 0.0131 0 0.013137 * <other> 0.04 0.0113 1 0.011286 <anon>:...y/Perl/5.10.0/CGI/Fa +st.pm:18 0.01 0.0029 1 0.002868 <anon>:...ibrary/Perl/5.10.0/C +GI.pm:27 0.01 0.0025 71 0.000035 main::dependency_checks

And I tried using Devel::FastProf, which gave me this output (-p was used)(Again, only a portion):

/System/Library/Perl/5.10.0/CGI.pm:867 22.53873 1: [at line 10 inside +eval] ??? /Library/WebServer/CGI-Executables/mytestsearch2011.cgi:139 7.02436 11 +7492: next unless ($sentblock =~ /\b$verbform\b/i); ##Ensure the sent +ence contains the searchkey /Library/WebServer/CGI-Executables/mytestsearch2011.cgi:123 0.86092 51 +9: while (my $sentblock = <FILE>) /System/Library/Perl/5.10.0/File/Find.pm:895 0.61021 150: @filenames = + readdir DIR; /Library/WebServer/CGI-Executables/mytestsearch2011.cgi:134 0.22259 29 +373: $sentence =~ s/, / /g; /Library/WebServer/CGI-Executables/mytestsearch2011.cgi:131 0.08909 30 +413: next unless ($sentblock =~ /\[sent. (\d+) len. \d+\]: \[(.+)\]/) +; ##Remember, talking about the whole block here /Library/WebServer/CGI-Executables/mytestsearch2011.cgi:126 0.05230 30 +413: if ( $sentblock =~ /file:\s(\S+)\.txt/) ##Could change to get ju +st chpt number/word /System/Library/Perl/5.10.0/File/Find.pm:940 0.03863 3625: $sub_nlink += (lstat ($no_chdir ? $dir_pref . $FN : $FN))[3]; /Library/WebServer/CGI-Executables/mytestsearch2011.cgi:27 0.03166 1: +require '/Users/jon/Desktop/stanford-postagger-full-2011-04-20/verbTe +nseChanger.pl'; /Library/WebServer/CGI-Executables/mytestsearch2011.cgi:132 0.03129 29 +373: $sentencenumber = $1; ## $1 refers to first set of paranthesis i +n the last match

Hope you can help.

Replies are listed 'Best First'.
Re: Benefiting from Devel::Profile or Devel::FastProf
by locked_user sundialsvc4 (Abbot) on Jun 22, 2011 at 15:42 UTC

    AUTOLOAD is a convenient-but-expensive mechanism.   (See, e.g.:   perlsub,   AutoLoader,  or perldoc perltoot.)

    Ideally, you would structure your program so that it is not necessary.   (But, maybe you can’t do that in this case.   Okay.)

    Furthermore, you might want to take a close look at the Plack family of packages, which provide a very good “universal compatibility-layer / grease-joint” mechanism for running your web programs in a variety of environments, particularly including “FastCGI,” and for switching the app more easily between different deployment strategies should you in the future identify the need to do so.

    One reason why I make this suggestion is that, if you decide that autoloading really is the most expeditious way to structure this application (or that the notion of changing a crufty old piece of code is just too risky), these deployment strategies might allow you to make the existing application persistent for a little while, so that the cost of autoloading can be amortized over many successive web-requests instead of being paid by each and every one.

      As for what to do about AUTOLOAD ... well, that depends entirely upon your application.   Generally speaking, I don’t like computer programs to do much of anything “automatically,” especially not at the behest of an (untrusted, of course) end-user.   However, when dealing with a legacy application that’s now in-service, you do not always have all of the practical options that you would like.   If you know why the auto-loading is happening and if you are comfortable with the overall security of the present design (neither of which I am specifically commenting-on here ...), then the most practical strategy might well be to simply try to make the million-pound elephant that’s paying the bills just a little bit more comfortable.

      Plack is, as you can see, a family of related packages ... the reference implementation of the so-called PSGI protocol.   It does have a variety of “canned” interfaces, including ones for CGI, which allow you to adjust your existing application with a minimum of effort – for example, to run under Apache’s mod_fcgid or mod_fastcgi.   (Under the latter, which I am more familiar with, the so-called “dynamic server” option is basically designed for just this purpose, and Plack more-or-less can supply all the rest of the necessary glue.

      Okay, Plack looks like it will help... (does PSGI need to replace CGI), however, I am confused about one thing still. Was the AUTOLOAD caused because my code called an undefined subroutine? Is it the values passed in, or the subroutine itself not loaded (would Autoloader solve it).

      I am pretty sure I can restructure my code so it is unnecessary, but don't even know where to start. Could you point me in the right direction, or do I need to supply my code?

      Thanks a lot

      p.s. your perltoot link just sends to super search, not the perldoc (if that's what you were going for)

        AUTOLOAD is an interesting gamble. It allows a faster module load time while slowing the 1st invocation of a number of a module's subroutines/methods. If you use few of those subroutines, the gamble pays off. If you use many.....

        In your case, CGI::AUTOLOAD is the from the CGI module. To fix the situation you can either call fewer of the subroutines (probably not realistic) or do as the above response suggests and make those 1st invocations far fewer in percentage by allowing each instance of your perl program to handle more than one request.

        To see perltoot you can use perldoc perltoot on the command line, or ask for perltoot on perldoc.perl.org . perl comes with a wealth of documentation.

        TJD

Re: Benefiting from Devel::Profile or Devel::FastProf
by 7stud (Deacon) on Jun 23, 2011 at 01:18 UTC
    I am confused about one thing still. Was the AUTOLOAD caused because my code called an undefined subroutine?

    Yes. But that isn't your fault. If a perl package tells you that you can call a certain method, then you can call it. How the package handles that method call is another matter. The package developer may define the method, or he may omit the method and define a method named AUTOLOAD to handle the method call. Letting AUTOLOAD handle the method call is less efficient but it can be much more flexible.

    use strict; use warnings; use 5.010; sub AUTOLOAD { our $AUTOLOAD; say "A method named $AUTOLOAD was called."; return 10; } my $result = calculate(); say $result; --output:-- A method named main::calculate was called. 10
    =====
    use strict; use warnings; use 5.010; sub AUTOLOAD { our $AUTOLOAD; my (undef, $greeting) = split '_', $AUTOLOAD; say $greeting; } say_goodbye(); say_hello(); --output:-- goodbye hello