Surely there must be an easier way?

Why? Looks pretty easy. For example:

my @a_indices = (1,3,5);
my @b_indices = (0..9);

@b_indices[@a_indices] = ();
@b_indices = grep defined, @b_indices;

use Data::Dumper;
print Dumper \@b_indices;

__END__
$VAR1 = [
          0,
          2,
          4,
          6,
          7,
          8,
          9
        ];
[download]