You can save a bit of memory by first grepping for defined elements inside each array, and then combining them:

my @flat = map { grep defined, @$_ } @{$aoa};
[download]

That way the "big" list with all the elements, including the undefs, isn't fully created in the first place.