Your main problem is here:

foreach my $key (keys %HoA) {
    foreach my $value (values %HoA) {
        my @Array = $value;
[download]

For each key in your hash, you're looping through all the values in your hash. You want to loop through the keys, and for each one, average the values in the particular array that it points to. Also, in the last line above, assigning a scalar to an array will simply make the scalar the first element of the array. Since your scalar is an array reference, you want to dereference it to get the array it points to. Which gives you these lines instead of the above:

for my $key (keys %HoA){    # loop through the keys
    my $value = $HoA{$key}; # get the array ref for this key
    my @Array = @{$value};  # dereference it to get the array
[download]

You can shorten that up and eliminate a step or two once you understand it, but that's the process broken down into small steps.

One last thing: no need to instantiate $average outside your loop, since you only use it inside the loop. Better to instantiate it inside, in the narrowest possible context.

Edited: Had a typo, $Array instead of @Array.