joealba asked:

Does anyone else have some good ideas on how to make this program a little more robust in its search, without returning too many misses?

Yes. Forward all of the images to me and I'll let you know if they are innapropriate.

Now, surprising as it may seem, I don't know a lot about the online porn industry (yet another area of future research, I suppose), but I am suspecting that they probably will redirect from the innocuous names to the suspect ones. Thus, you'll probably want to check for redirects. If you're using LWP::Simple, be careful. For example:

perl -MLWP::Simple -e "getprint(q|http://www.ovidinexile.com/|)"
[download]

The above code will print out HTML for a frameset. However, if you use Rex Swain's HTTP viewer, you discover that you are redirected to my real home page. I think a redirect should definitely be something you want to flag, even if the Russian words on the new site don't trigger your regexes :)

Cheers,
Ovid

Join the Perlmonks Setiathome Group or just click on the the link and check out our stats.

NICE! I like the idea of returning every link that results in a redirect. Thanks, Ovid!

So I should set up an HTTP::Request, pass it to an LWP::UserAgent, and check the $response->{_rc} response code? Or is there an easier way?

Something like this may help...

if ($res->is_success) {
   #Normal retrievel of content stuff
}else{
   #check for redirects
  if ($res->code() =~ /30[12]/){    #redirect codes (temp/perm)
     #grab the location
     my $remote_cgi = $res->header('Location');
    {
       # Some servers erroneously return a relative URL for redirects,
       # so make it absolute if it not already is.
       local $URI::ABS_ALLOW_RELATIVE_SCHEME = 1;
       my $base = $res->base;
       $remote_cgi = $HTTP::URI_CLASS->new($remote_cgi,$base)->abs($ba
+se);
    }

}else{
    # Request failed normaly, broken link
}
[download]

Where $res is your result object

---If it doesn't fit use a bigger hammer

Note: I'm not saying that you're wrong. At all.

I'd be inclined to take a different tack on this.

Script to check the last modfied date/time on external links, and if changed since last checked give the list for a human to check.

Of course, there are issues with getting the last modified accurately, but I imagine that they're more solvable than parsing for content.

Perhaps each page has a certain string you can check for to make sure it's unchanged?

Hope the different viewpoint helps.

Malach
So, this baby seal walks into a club.....

That's another good idea, but most (if not all) of our external links will be updated quite often. We just don't have the manpower to check every link every time it is updated.

Besides... why have people do the work that a few well-planned regexps can do? :) Thanks, though!

Well... you could however have your script only check those pages that changed... and as a safety net check pages that haven't reported changes on a less frequent basis... to make it so your script doesn't run forever.

                - Ant
                - Some of my best work - (1 2 3)

It sounds like you have thought out the problem pretty well. You already make reference to using the modules and elements I'd normally recommend. And I'm glad to hear that this isn't a case of knee-jerk reactionism, that you plan on having the "hits" reviewed by human eyes before axing them.

I'd say charge forward with the plan you have, and let us know if you run into any difficulty code-wise.

Plus, I'm tempted to give you a ++ vote just for the sheep reference in your "naughty sites" RE :-).

--rjray