For that degree of difference you are getting very close to meaningless values. Consider:

use Time::HiRes qw(time);

my @results = ();

push @results, [TimeIt()] for 1 .. 10;
shift @results; # First result is polluted by startup processing
@results = sort {$a->[0] <=> $b->[0]} @results;
printf "Out: %.5f  In: %.5f diff: %+.6f\n", @$_, $_->[1] - $_->[0] for
+ @results;
printf "Out delta %f\n", $results[-1][0] - $results[0][0];
@results = sort {$a->[1] <=> $b->[1]} @results;
printf "In delta %f\n", $results[-1][1] - $results[0][1];


sub TimeIt {
  my ($i);
  my $start = time;
  {
    my $x;
    for $i (1..1e6) {
      $x = 1;
    }
  }

  my $end = time;
  my $deltaOut = $end - $start;
  
  $start = time;
  for $i (1..1e6) {
    my $x = 1;
  }
  $end = time;
  
  my $deltaIn = $end - $start;

  return $deltaOut, $deltaIn;
}
[download]

For one run prints:

Out: 0.01120  In: 0.01609 diff: +0.004889
Out: 0.01121  In: 0.01474 diff: +0.003521
Out: 0.01123  In: 0.01501 diff: +0.003781
Out: 0.01126  In: 0.01509 diff: +0.003836
Out: 0.01128  In: 0.01494 diff: +0.003654
Out: 0.01135  In: 0.01491 diff: +0.003559
Out: 0.01148  In: 0.01524 diff: +0.003761
Out: 0.01291  In: 0.01497 diff: +0.002066
Out: 0.01358  In: 0.01504 diff: +0.001458
Out delta 0.002376
In delta 0.001356
[download]

Note that the difference between the fastest and slowest "outside the loop" times was close to the difference between the individual test "inside loop" - "outside loop" times. This changes somewhat from run to run, but ultimately at this level your code's performance often depends more on external factors than the code itself.

Perhaps also note that my results are about a factor of two faster than your results so maybe you just need to buy a faster computer? That's not meant to be mean or snarky, but just to point out that micro optimisation of this sort can usually be beaten by either getting faster hardware or improving algorithms. You may be interested in Youtube - Matt Parker: Someone improved my code by 40,832,277,770% to see the extreme version of this comment!