In the example I mentioned I was around to witness it _was_ a bank and it was the online banking application for business accounts. They spent somewhere around 2 million on the software and planned on spending about $250,000 on the hardware. Turns out they had to spend nearly 6 million on hardware just to get the thing to support the current customer base. Meanwhile we (the internal developers) had the system supporting the personal banking app working on half the hardware, supporting 20x the sessions... Yes, the bank had the money to purchase the additional hardware. Yes they sued the development company for delivering faulty software. Yes, probably far less than 1 million spent on efficiency tuning would have let them save over 5 million on hardware. Just because a company has the money to throw at a problem doesn't mean they should spend it. In this case they had to go that route because of a hard launch date tied to national advertising campaigns. (God I hated those, they'd sometimes tell us of a campaign to start in a few days and then request about 3 weeks worth of work to support the campaign.)

The systems i've dealt with have hundreds or thousands of tables in the backend with tens of thousands of columns and millions to billions of rows. One of them automated the companies knowledge and processes to the point where we could hire $12/hour operations staff instead of $50/hour professionals as the knowledge was manged by the system not the people.

Another company i've seen with efficiency problems is one of the larger ERP application providers recently purchased by a fortune 500 database company. It is massively complex software and I recall hearing the company's tech snickering, leaving a requirements session saying things like "that screen they want to draw will take 20 minutes to generate" (a screen to be used regularly by people in bank branches to service customers).

Does the under 25kLOC count the code from CPAN modules used? The one huge efficiency bugaboo I encountered in a CPAN module " DBIx::Class" (actually it was in redhat's version of perl5.8.8 was the culprit, but it was DBIx::Class that was exercising the slow section of perl). With the problem which was a bugfix of only a handful of lines perl/DBIx::Class got an order of magnitude slower for certain operations. Operations that would definitely affect site performance. If memory serves that bug also affected things other than DBIx::Class but i can't recall what. https://bugzilla.redhat.com/show_bug.cgi?id=196836

I should mention again that I don't value efficiency over flexbility and clean code. But time and time again i've seen the value of having an efficiency mindset. Personally I prefer to start flexible and optimize where necessary.

Consider omething simple like this:

open(FILE, "<billionlinelogfile.log");
my @file = <FILE>;
close(FILE);
foreach my $line (@file) {
  do stuff...
}
[download]

open(FILE, "<billionlinelogfile.log");
while(<FILE>) {
  do stuff...
}
close(FILE);
[download]