The problem must be with your input data, as I get a match with the following test cases :

sub check {
  $_ = "sendmail[[text to be captured from sendmail],relaying denied";

  (/sendmail\[.*?\[(.*?)\],.*?Relaying denied/i) and return ($1);
  (/gnu-pop3d\[.*?refused connect from (.*?)$/i) and return ($1);
  (/cucipop\[.*?authentication failure\s.*?\s(.*?)$/i) and return ($1)
+;
  
  return "No match found";
};
print check,"\n";
[download]

and also with

sub check {
  $_ = "sendmail[[text to be captured from sendmail],relaying denied";

  if (
    (/sendmail\[.*?\[(.*?)\],.*?Relaying denied/i) ||
    (/gnu-pop3d\[.*?refused connect from (.*?)$/i) ||
    (/cucipop\[.*?authentication failure\s.*?\s(.*?)$/i)
  ) { return ($1) };
  
  return "No match found";
};

print check,"\n";
[download]

The matches are exactly the same as your matches. My guess is, that you've got an extraneous \[ sprinkled in your sendmail RE.

perl -MHTTP::Daemon -MHTTP::Response -MLWP::Simple -e ' ;    # The  
$d = new HTTP::Daemon and fork and getprint $d->url and exit;#spider
($c = $d->accept())->get_request(); $c->send_response( new   #in the
HTTP::Response(200,$_,$_,qq(Just another Perl hacker\n))); ' #  web
[download]