Curlies can be used for disambiguation:

$input =~ s/<${imlre{$_}->[1]}[^>]*>//g ;
[download]

When dealing with HoA, HoH, AoA and AoH, the arrays are optional between the indexes.
$imlre{$_}->[1]
is the same thing as
$imlre{$_}[1]

When Perl sees
$imlre{$_}->[1][^>]
it thinks you mean
$imlre{$_}->[1]->[^>]
rather than a HoA followed by a regexp. Between the square brackets, an index is expected. Since ^> is not a valid perl expression, you get an error.

Update: Rearranged explanation for clarity.