When loading large volumes of data, a little care can go a very long way.

• If I load an array with 9e6 values your way it requires 905MB(*) of ram to complete the process:

  [11:30:52.92] C:\test>perl -E"
  
  @a=1..9e6; 
  
  say for grep /$$/, `tasklist`"
  perl.exe                      5632 Console                    1    937
  +,368 K
  
  
  [11:30:56.08] C:\test>
  [download]

  (*YMMV. I'm using 64-bit Perl. It will be less on 32-bit.)

• But, with just a little effort, I can populate that same data using under 300MB:

  [11:31:01.54] C:\test>perl -E"
  
  $#a=9e6; $a[$_-1]=$_ for 1..9e6; 
  
  say for grep /$$/, `tasklist`"
  perl.exe                      3396 Console                    1    292
  +,188 K
  
  [11:31:04.24] C:\test>
  [download]

  And if you look closely at the timestamps, it even runs around 20% faster.

• And if I'm really pressed for space, I can cut that down to less than 40MB with very little extra effort or time cost:

  [11:47:33.84] C:\test>perl -E"
  
  $a=chr(0); $a x=9e6*4;substr($a,4*($_-1),4,pack'N*',$_) for 1..9e6; 
  
  say for grep /$$/, `tasklist`"
  perl.exe                      8028 Console                    1     40
  +,308 K
  
  
  [11:47:37.79] C:\test>
  [download]

  Accesses will be a tad slower, but no so much as to take me anywhere near 24 hours to sum and count a few million data points.

All commands wrapped for clarity!

And the same or similar techniques can be used for most every aggregate population task. It is just a case of knowing when to use them.

Most of the time we don't bother because our data sizes are such that it isn't worth the (small) effort; but it behooves us to know when the small extra effort will pay big dividends.

As for the blogger; quite why he feels the need to load all his DB-held data into his program in order to do bread and butter SQL queries is beyond me.

Whilst I don't entirely disagree with the premise that there are times when Perl isn't the right choice; making the fundamental error of pulling all his DB-held data into perl in order to perform processing that he actually describes as "all you need is sum(field), count(field) where date between date1 and date2,", just makes me doubt the veracity of his conclusions.