6-7 seconds processing is an awesome amount of time unless your machine is dog slow. First have a look with top to see if you are running out of RAM and getting into the swap space this will really kill you for speed. If so add more RAM (its cheaper than hacking time). Perl loves RAM and so do DBs for that matter. You really can't throw too much RAM at either.The more RAM a DB has available the more data and indexes it will cache. Most of my servers have at least 2GB.

You must be doing a lot of post processing on the DB output to take 6-7 seconds. Don't. Sorting, Grouping, Joining, Selecting are all tasks that should be done at the DB level (fast and in C) not the application level (slower in Perl). This of course may entail major recoding. If you are only displaying X results then add a LIMIT clause so you only get that many rows - quicker to get, less to post process for a minor change.

You must have a lot of loops to be consuming this much time. Rewrite them to either exit as soon as they can ie:

# slow
print "Found" if grep{ /$find/ } @list

# always faster and uses less memory
for (@list) {
    do{ print "Found"; last } if /$find/;
}
[download]

or preferably do it at DB level as noted.

On the subject of loops you can spend memory and gain speed by building hash lookup tables rather than iterating over lists.

If you are doing sorts that can't be done on the DB with an ORDER BY clause you may need a Schwartzian Transform depending on the application.

If you are calling external processes or including dynamic content you will get benefit from caching static results, updating them as required, and using these.

PerlRun is an option but all it really does is save a bit of time on the compile and load which is apparently not your issue given the time breakdown. It will help and is the simplest option if the code will run OK under it.

cheers

tachyon

s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

Though I agree about the RAM thing, using DBMS functions to process the data is not always faster. I've found SELECT DISTINCT to be, on occasion, many times slower than feeding the output of a simple SELECT into a hash.

This may have been an unusual case. I was retrieving distinct values of a single column from a table with about 125,000 rows from Empress, which isn't that much of a mainstream system. Still, I seem to remember using the Perl hash was more than 20x faster that letting Empress do the SELECT DISTINCT thing.

--
bowling trophy thieves, die!

Or buy a better product.

Abigail

I apologize for not describing the profile. Out of the ~10 seconds, the breakdown is something like the following:
• 1 second on startup - database connection, etc.
• 1-2 seconds on database work
• 6-7 seconds on "processing"

It's the "processing" step that causes so much trouble, since we can't make major modifications to the code.

And we have definitely thought about buying a better product, or writing one in house. Ugh.

You can probably get it to work under PerlRun. That would completely eliminate the startup time. It won't help with the processing time though. For that, you really have to profile things and change code. The only generic piece of advice I can offer is to run perl 5.6.1 compiled with no threads, since it seems to have better performance than later perl versions.

Incidentally, FastCGI and SpeedyCGI have the same limitations as mod_perl when running poorly written code. PerlRun does a better job than either of them do at trying to make things work.

Well, then you can save at most 1 second by changing the program to be both mod_perl (or something similar) aware, and by using some database connection pooling mechanism. You might be able to save a little on the database work by using different indices or a different layout. But that also saves you at most 2 seconds.

The majority of the possible savings will be from changing the code. But you say you can't make major modifications. There isn't a magical wand you can wave that instantly makes all programs faster. So, I'd say, your task is impossible.

Abigail

Profile. See Devel::DProf. Concentrate your efforts at the bottlenecks. If you find slow bits that you can't see how to improve, post them.

If your reading files, templates and the like from disk, create a ram drive and load them from there.

NB: This is highly speculative! If there are truely lots of globals in use, you might get a little from making them lexicals by declaring them at the top of each file using our, assuming your using a version of perl that supports this.

Beyond that, your the man who can see the code. What's it doing that takes so long? PROFILE:)

you might get a little from making them lexicals by declaring them at the top of each file using our

It my understanding from this doc that our creates globals. The perl561delta.pod document mentions that they "... can be best understood as a lexically scoped symbolic alias to a global variable" (which I must confess did not assist my understanding one iota :-) ). But 'our' variables do create symbol table entries with the associated typeglob memory footprint overheads.

Hence the speculation.

It was pointed out to me that, in some circumstances, that using lexicals rather than globals is quicker. One explaination I saw, but cannot now find, is that it is quicker to find a lexical than a global? I took this to be something to do with the fact that the compiler has to know where a lexical is at compile time, and effectively hard codes the 'location' into the optree, but that globals are 'found' at each time, at runtime. The technical detail my be wrong but this simplified imagery "works for me" ... until a better explanation is available.

How far this case extends I have never tried to assertain as I have found very few case where I use globals, and in the few cases I do, they are never citical to performance, but the the OP seemed to be looking for 'simple' measures that might help. Adding use strict at the top of a module will rapidly spit out the names of the globals. Adding one line, our ($a, $b, $this, $that); with an appropriate comment seem a simple enough change to at least make it worth trying, given the lack of other information and the restrictions imposed by the OP's question.

---

Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller

I have profiled, more with timestamping than anything else. Sorry for not stating this in my original writeup.

I guess this is more a question of what to do if you have profiled, proven that the application is at fault, and probably won't be able to run it in mod_perl. Anything but throw more powerful processors at it or buy/write a new application?

Thanks for your input, BrowserUk and Abigail-II.

Sometimes, but only sometimes, small changes to the code can reap large rewards. As an example, if the application truely is Perl 4 code, then I believe (but could be wrong) that Perl 4 didn't support hashes. Assuming for the moment that is true -- I'll be quickly corrected if it isn't:) -- and (for example) the application does any sort of lookups into arrays of data using grep, then changing the array(s) being search linearly to hash(es) could have a dramatic effect without too much effort.

I'm not really sure I understand the reluctance to modify. You said this is because you hope to upgrade to the next version sometime in the future. If you make changes now, how does that stop you upgrading? The only reason refactoring the code (ie. not changing what the code does, but only the way it does it) would impact your upgrade, is if you discovered that the later version wasn't as efficient as your modifed version. In which case you would have very strong grounds for requesting that your changes be fed back into the latest version before you took delivery. The supplier might even thankyou for it and reduce your bill (some chance:). Their customers almost certainly would thankyou.

At the end of the day, if you change nothing, nothing will change. If you can't change the code, then you already know the other options. More memory, a faster processor, harddisks etc.

I think you already knew this though, so it begs the question, what were you hoping for?

Oh! You have already done the Monk's ritual haven't you?

---

Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller

Well, your major concern seems to be avoiding large changes to ensure that you can hope to upgrade - have you tried talking to the vendor, to see if either they are already working on speed improvements, or would consider rolling your changes into any revision of the product?

Hugo

I doubt you'd see a huge speedup by hacking the code to eliminate globals and uses of local where my captures what was intended. I also doubt that there would be such a thing as a smooth upgrade from this product to one that adheres more to the Perl5 or (by the time it comes out?) Perl6 way of doing things that didn't simply preserve or transform the underlying data. The code involved in an upgrade is likely to be all new, assuming it provides performance boost of the order you're seeking. As to the mod_perl approaches, they won't likely help unless starting the Perl interpreter is the problem (if it takes 1 second to start the interpreter, and 8 seconds to gather and organize the data, and another second to generate a page now, under PerlRun it might go down to .2 seconds startup time, 8 seconds to collate, and a second to generate the page ¹. The best thing you could do is profile your app to see where it's spending the most time to get an idea of where you might start optimizing. There is no point spending several programmer-weeks optimizing a little routine that accounts for .02% of the time the script runs (be sure to point out that time you spend on this is time not spent on other stuff -- that must be taken into account when weighing any of the options here). Search for "profiling" on CPAN; you might start by checking out Devel::DProf although I'm sure there are profiling gurus around here who can help you make a good choice.

Best, and this is hard to say much about in a vacuum, is to look for places where it's manifestly inefficient, although this is likely to be hard to track down. If you can, get the system to cache frequently accessed or hard to calculate data / generated pages. If it's doing the same beast of a computation for each request, and the results of that computation change relatively slowly and are valid for a wide range of users, then a cache will really speed it up. Maybe your solution could be as simple as putting the web application behind a caching proxy, once you figure out how to cache to best match your desire for up-to-date personalized data against performance concerns.

As to inefficient algorithms, a really obvious case would be a subroutine that searches an array that's called a lot which looks like

# ok, yeah, this is perl5ey
sub in_array {
   my $foo = shift;
   return grep $_ eq $foo, @global_array;
}
[download]

In the end, the thing to do may be to upgrade the software and spend your time figuring out how to migrate the data.

my $out='';
for (0..1000)
{
  $out .= "$_"." bottles of beer on the wall\n";
}
print $out;
[download]

for (0..1000)
{
  print "$_"," bottles of beer on the wall\n";
}
[download]

package Print;
require Exporter;
@ISA = qw(Exporter);
@EXPORT_OK = qw(Print);

our $sout='';

sub Print
{
  $sout .= "@_";
}

END
{
  print $sout;
}
1;
[download]

use Print qw (Print);
for (0..1000)
{
  Print "$_"," bottles of beer on the wall\n";
}
[download]

It is good practice to create some output for the user to look at early in your program, then buffer up the rest of the output that is slow to generate.

It should work perfectly the first time! - toma

I'd personally say go for it and make sweeping changed to get the speed out. If, as you say, the application feels like it was written for Perl4 then hoepfully any major future upgrade will involve a general update, and so won't need your changes done since it'll be fast enough as is.

If you can arrange for your "processing" to be done at a db level you may be able to simplify the amount of perl-based processing you do. For example you might be able to use a temporary table to do things. Or as mentioned above grouping will be faster in the db.

Can you really not provide any finer resolution on the amount of time taken in the subprocesses of the "processing" step? You should be profiling everything otherwise how are you going to find where the bottleneck is in the code?

I did not disclose the problem domain because I did not want to harm the company we purchased the product from.

Thanks for the suggestion on serving static pages. It's an angle I hadn't really considered before. We could probably come to a compromise on the frequency of page updates, like once every 30 minutes.

Update: Clarified wording about the problem domain.

I agree with Jaap - tell your boss that you can write something better in 3 months, which would be the amount of time you'd take to speed it up and deal with the first upgrade. Heck, you could upgrade it, then sell your upgrades back to them!

------
We are the carpenters and bricklayers of the Information Age.

Don't go borrowing trouble. For programmers, this means Worry only about what you need to implement.

Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.