Kudos to you for wanting polite bots. The problem with getting LWP::RobotUA to play nice with WWW::Mechanize is that they both are subclasses of LWP::UserAgent. By itself, WWW::Mechanize does not consult the /robots.txt file, but you can instead use WWW::RobotRules. Here is a working example that tries to grab two files from my server: 
use strict;
use warnings;

use WWW::Mechanize;
use WWW::RobotRules;
use LWP::Simple;

my $SITE  = 'http://www.unlocalhost.com';
my $rules = WWW::RobotRules->new('bot/1.0');

my $robot_url  = "$SITE/robots.txt";
my $robot_data = LWP::Simple::get($robot_url);
$rules->parse($robot_url, $robot_data) if $robot_data;

for ('disallow.txt', 'allow.txt') {
   my $url = "$SITE/$_";
   if($rules->allowed($url)) {
      my $mech = WWW::Mechanize->new;
      $mech->get($url);
      print "$url:\n", $mech->content;
   }
   else {
      print "$url:\ndenied\n";
   }
}

[download]
There might be a better way though ... ahh, how about "WWW::Mechanize::Polite"? 
package WWW::Mechanize::Polite;
use base 'WWW::Mechanize';
use WWW::RobotRules;

sub new {
   my $self = shift->SUPER::new(@_);
   $self->{robo_rules} = WWW::RobotRules->new($self->agent());
   return $self;
}

sub parse_robots {
   my ($self,$url) = @_;
   $self->get($url);
   $self->{robo_rules}->parse($url, $self->content);
}

sub polite_get {
   my ($self,$url) = @_;

   if ($self->{robo_rules}->allowed($url)) {
      $self->get($url);
   }
   else {
      undef $self->{content};
   }
}

1;

[download]
And a client script: 
use WWW::Mechanize::Polite;

my $SITE = 'http://www.unlocalhost.com';
my $mech = WWW::Mechanize::Polite->new;
$mech->parse_robots("$SITE/robots.txt");

for ('allow.txt', 'disallow.txt') {
   my $url = "$SITE/$_";
   $mech->polite_get($url);
   print "$url:\n", $mech->content ? $mech->content : "denied\n";
}

[download]
And if i didn't just reinvent a wheel, you might be seeing this on the CPAN. ;)

jeffa

L-LL-L--L-LL-L--L-LL-L--
-R--R-RR-R--R-RR-R--R-RR
B--B--B--B--B--B--B--B--
H---H---H---H---H---H---
(the triplet paradiddle with high-hat)

In reply to WWW::Mechanize::Polite ? by jeffa
in thread Using URI::URL by mkurtis

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.