Is there no way to insert these values directly into those array refs with push or something else?

Well, you could directly assign each value to the next element off the end of each array with code like this:

($a[0][@{$a[0]}], $a[1][@{$a[1]}]) = unpack("x$i A1 A1", $b) ;
[download]

But that's horrible!