subcall( $M[0] );

Assuming that  @M is initialized as shown in Re^2: Arrays in arrays, how to access them, you get, in this instance, an array reference passed to the  subcall() function because that's what element 0 of the  @M array is.

subcall( \@{M[0]} )

In this instance (assuming it's fixed to
    subcall( \@{$M[0]} )
by adding the missing  $ sigil), you have, working from the inside out:

• an array reference:  $M[0]
• de-referenced to an array:  @{$M[0]}
• to which a reference is taken:  \@{$M[0]}
• this reference then being passed to a function:  subcall( \@{$M[0]} )

In this instance, you're simply wrapping the access to  $M[0] in a redundant pair of complementary operations — and maybe in this case the compiler would even be smart enough to optimize away the redundancy! It's as if you had asked "Why do the two function calls
    func($x);
and
    func($x + 1729 - 137 - 1729 + 137);
both pass the same value to the function?"

Ah, but if I use
subcall ( @{\@{$M[0]}} )
...

If you use
    subcall ( @{\@{$M[0]}} )
you extend the process of the second instance above to de-reference the array reference created (redundantly) by  \@{$M[0]} and pass an array (in the example, the  @a1 array) to the  subcall() function, which is apparently what it takes to make that function happy.

>perl -wMstrict -le
"my @a1 = qw(a b c);
 my @a2 = qw(p q r);
 my @a3 = qw(x y z);
 my @M = (\@a1, \@a2, \@a3);
 ;;
 print $M[0];
 print \@{$M[0]};
 ;;
 print 'array references are the same' if $M[0] == \@{$M[0]};
 ;;
 print qq{-@{\@{$M[0]}}- -@{$M[0]}-};
"
ARRAY(0x43281c)
ARRAY(0x43281c)
array references are the same
-a b c- -a b c-
[download]

(Also see perlreftut.)