Sheesh, part of it was the stupid -ggdb flag. Still there's a 10 MB penalty.

  PID COMMAND      %CPU   TIME   #TH #PRTS #MREGS RPRVT  RSHRD  RSIZE 
+ VSIZE
17233 perl         0.0%  0:03.03   1    12    46  13.2M  1.42M  14.4M 
+ 33.3M 
17232 perl         0.0%  0:05.40   1    12  1176  14.1M  7.79M  20.2M 
+ 43.8M
[download]

Still experimenting...