Your "Output i'd like to have" for "Track 1" doesn't include the value from the second segment. You didn't indicate any exclusions so I've included this in my code below. I've used floating point numbers for the averages instead of integers.

my $VAR1 = [ ... ]; # Dumper output you posted (s/VAR1 = = [/VAR1 = [/
+)

print qq{Average elevation\n};

for my $track (@$VAR1) {
    print q{  }, $track->{name}, q{ = };

    my $track_ele_total = 0;
    my $track_ele_count = 0;

    for my $segment (@{$track->{segments}}) {
        for my $point (@{$segment->{points}}) {
            ++$track_ele_count;
            $track_ele_total += $point->{ele};
        }
    }

    printf qq{%06f\n} => $track_ele_total / $track_ele_count;
}
[download]

Which outputs:

Average elevation
  Track 1 = 69.750000
  Track 2 = 61.000000
[download]