...the queue that resides in $mcvpservers->{$server}->[2]. Do I have to push to that reference somehow?

If that's an array reference, you probably don't want

    my @queue = $mcvpservers->{$server}->[2];
[download]

but rather

    my $queue = $mcvpservers->{$server}->[2];
    ...
    push @$queue, ...
[download]

Update: as you have it, your array ref (queue) will be assigned to the first element of @queue, but whatever you push onto @queue will be added outside of what is pointed to by the array ref. Consider

use Data::Dumper;
    
our @queue = (qw(stuff in queue));
my $server = "name";
$mcvpservers->{$server}->[2] = \@queue;
# ...

my @queue = $mcvpservers->{$server}->[2];
push @queue, "morestuff";

print Dumper \@queue;
__END__
$VAR1 = [
          [
            'stuff',
            'in',
            'queue'
          ],
          'morestuff'
        ];
[download]

I'm not sure that's the problem, but if it is, how do I write $#queue+1 and $#{$queue[0]}+1 with array refs instead? I want to check if the queue is empty and the length of it.

how do I write $#queue+1 and $#{$queue[0]}+1 with array refs instead?

$#$queue+1
$#{$queue->[0]}+1
[download]

or simply

scalar @$queue          # number of elements
scalar @{$queue->[0]}
[download]

(you don't need the scalar if it's in scalar context anyway, like with if (@$queue) )

Presuming the first element of the referenced array holds another array ref, that is (not sure what you intend to keep there...).