Not only is a factorial calculator not a good candidate for recursion, Perl (currently) doesn't do tail-recursion optimization, which would fix your memory leak.

I'm also trying to figure out how your two examples are similar. The first defines new anonymous subs, but the second only defines one. (The additional subs are defined because you're calling the function definer over and over.) Try the following:

use strict;             # leak occurs whether strict is used or not
 
my $fact = build_sub();
while(1) {
  print &$fact(5), $/;
}
 
sub build_sub {
  my $factorial_sub_ref;
 
  $factorial_sub_ref = sub {
    my ($number) = @_;
    if ($number < 2) { return 1; }
    return($number * &$factorial_sub_ref($number - 1));
  };
 
   return $factorial_sub_ref;
}
[download]

Also, factorials are multiplied, not added.