Keep in mind the point of the article: It's impossible to do URL normalizing well. They've even missed two items that need normalizing (but they might be in the linked spec):

1) The order of the arguments in a GET:
.../script.cgi?a=b&c=d vs
.../script.cgi?c=d&a=b

2) The domain name:
example.com vs
example.com. vs
EXAMPLE.COM

Oh and IP addresses too:
10.0.0.1 vs
0x0A000001 vs
167772161

Some background - I am developing a scraper and it needs to know if it has scraped the page already - hence the normalization. It needn't be perfect, just good enough for all but the most arcane. Needs only work with http.

What about using the "last_modified" method in LWP? Keep track of it locally. When you access the page again, check the time it was modified and skip it if that time is not newer than what you've saved.

This idea is from "Spidering Hacks" (hack #16).

I haven't tried it but wouldn't URI and it canonical and eq methods work for you?

Update: Looks like URI will not normalize query params. Something like this should work (note, I did not check all cases - feel free to fix!)

!/usr/bin/perl -wd

use URI;

my $u1 = URI->new("http://www.perl.com/cgi-bin/script.cgi?a=b&c=d");
my $u2 = URI->new("http://www.perl.com/cgi-bin/script.cgi?c=d&a=b");

my $u1c = $u1->canonical;
my $u2c = $u2->canonical;

if( urlsEqual( $u1c, $u2c ) ) {
  print "equal\n";
} else {
  print "not equal\n";
}

sub urlsEqual {
  my( $u1, $u2 ) = @_;
  my( $q1, $q2 );

  # First try URI eq
  return 1
    if( $u1->eq( $u2 ) );

  # nope ... adjust query
  $q1 = $u1->query();
  $q2 = $u2->query();

  $q1 = join( '&', sort( split( /[&;]/, $q1 ) ) )
    if $q1;
  $q2 = join( '&', sort( split( /[&;]/, $q2 ) ) )
    if $q2;

  $u1->query( $q1 );
  $u2->query( $q2 );

  return $u1->eq( $u2 );
}
[download]

From what I saw, URI

• Lowercases the scheme.
• Lowercases the domain name. (1)
• Removes the port if it's the default. (2)
• Removes port fields consisting of just ':'. (3)
• Adds trailing '/' if no path or query is specified. (6, partial)
• Doesn't do (4), (5), (7) and (8), but easy to do.
• Doesn't do (9) and (10), but might not be possible.
• Doesn't set the path to '/' if no path is specified and a query is specified. (6, partial)
• Doesn't normalize IP addresses in to dotted form.
• Doesn't remove the trailing '.' from domain names, if any.
• Doesn't touch the query.

You can't expect a module called 'URI' to normalize CGI parameters. http://foo.com/bar?a=b&c=d and http://foo.com/bar?c=d&a=b are two different URIs. The fact the two different URIs are treated the same by the receiving server is outside of the URI realm.