Untested:

my @filtered_links = grep {
   my $response = $browser->get($_);  # Better: ->head($_)
   $response->is_success
   && (($response->content_type eq 'text/html') 
       || ($response->content_type eq 'text/plain'))
} @links;
[download]

Update: fizbin's is better, use head instead of get. Anyway, I'd retain the check on is_success for documentation and readability reasons.