To deal with this phenomenon, and to be a good network citizen, you should throttle your own requests to something reasonable, such as no more than three pages per second, or more kindly, one page per three seconds. Just call sleep(1) once in a while.

sub _extract_links {
    require WWW::Mechanize::Link;

    my $self = shift;

    my $p = HTML::TokeParser->new(\$self->{content});
    
    $self->{links} = [];

    while (my $token = $p->get_tag( keys %urltags )) {
        my $tag = $token->[0];
        my $url = $token->[1]{$urltags{$tag}};

        my $text;
        my $name;
        if ( $tag eq "a" ) {
            $text = $p->get_trimmed_text("/$tag");
            $text = "" unless defined $text;

        my $onClick = $token->[1]{onclick};
        if ( $onClick && ($onClick =~ /^window\.open\(\s*'([^']+)'/) )
+ {
        $url = $1;
        }
        }
        if ( $tag ne "area" ) {
            $name = $token->[1]{name};
        }

        next unless defined $url;   # probably just a name link or <AR
+EA NOHREF...>
        push( @{$self->{links}}, WWW::Mechanize::Link->new( $url, $tex
+t, $name, $tag, $self->base ) );
    }

    # Old extract_links() returned a value.  Carp if someone expects
    # this version to return something.
    if ( defined wantarray ) {
        my $func = (caller(0))[3];
        $self->warn( "$func does not return a useful value" );
    }

    return;
}
[download]

I ran in to the exact same problem but I got stung at 5 links. However putting in a sleep(5) did the trick for my particular case and I was off and running.

... or just use WWW::Mechanize::Sleepy instead.

--
b10m

All code is usually tested, but rarely trusted.