1) They're called "carets". They're not edible.

2) "it will only work if I broaden the regex by removeing the carrot and the scalar" Not so:

$_ = ':Admin!Admin@leet-A0B340B0.host.com PRIVMSG #channel :!quit';
$channel = '#channel';

if($_ =~ /^:(.*)!(.*) PRIVMSG $channel :(.*)$/){
   my $user = $1;
   my $host = $2;
   my $message = $3;
   if ($message =~ /^!quit$/){
      die "Quiting!\n";
   }
}

__END__
output
======
Quiting!
[download]

$_ is apparently not what you expect it to be.

3) Why use a regexp for to compare against a constant string?
$message =~ /^!quit$/
is not as good as
$message eq '!quit'

4) I bet
/^:(.*)!(.*) PRIVMSG (#\S+) :(.*)$/
is not as efficient as:
/^:([^!]*)!(\S*) PRIVMSG (#\S+) :(.*)$/

5) Why use $channel in the regexp? That's rather odd. I'd use:

if($_ =~ /^:([^!]*)!(\S*) PRIVMSG (#\S+) :(.*)$/){
   my $user = $1;
   my $host = $2;
   my $chnl = $3;
   my $message = $4;
   if ($chnl eq $channel) {
      if ($message =~ /^!quit$/){
         die "Quiting!\n";
      }
      ...
   }
}
[download]