This will do the job:

my %HoH = map { delete $_->{FirstNum} => $_ } @AoH;
[download]

Be aware that you will loose data if there is duplication of "FirstNum" values.