I’ll assume that the volume of data in question is not such that storing the data “in memory” is not a potential problem, and therefore that the possible suggestion of using SQL JOINs is not useful or necessary.

I would start by organizing the data into a tree, by group-number.   So you’d have something like this:

  %tree = (
    1 => {
           3 => [ ... ],
         },
    4 => {
           4 => [ ... ],
           6 => [ ... ],
           7 => [ ... ],
          },
    8 => {
           8 => [ ... ],
           11 => [ ... ],
         }
   ...
);
[download]

I admit that the requirement is not fully clear to me, especially when you start saying things like “the 3 array is removed and the 8 array is added.”   But even so, the general strategy ought to work, thanks in part to “references” making it possible for the same list to (if necessary) literally occupy more than one place in the tree at the same time.   You’d just wind up with more than one arrayref pointing to the same array-object, at different places within the tree, and Perl would see to it that the arrayref didn’t get garbage-collected until the last reference was consumed.