-- Randal L. Schwartz, Perl hacker

how about this:

#!/usr/bin/perl -w

use strict;

use Algorithm::Diff qw(diff LCS);
use HTML::TokeParser;
use LWP::Simple;

sub tokenize_url
{
    my $url = shift;

    my $content = get $url or die $!;

    my $p = new HTML::TokeParser(\$content);

    my (@tokens, $token);

    push @tokens, $token
        while (defined ($token = $p->get_token));

    \@tokens;
}

my @content = map { tokenize_url($_) } qw{
http://perlmonks.org/index.pl?node_id=32285
http://perlmonks.org/index.pl?node_id=32286
};

# hash tokens based on their text content
sub hash_token {$_[0][$_[0][0] eq 'T' ? 1 : -1]}

my @diffs = diff $content[0], $content[1], \&hash_token;
my @LCS = LCS $content[0], $content[1], \&hash_token;

my $largest = 0;

for my $hunk (@diffs)
{
    my (@deletions, @additions);

    for (@$hunk)
    {
        push @deletions, $_ if $_->[0] eq '-';
        push @additions, $_ if $_->[0] eq '+';
    }

    my $size = @deletions > @additions ? @deletions : @additions;

    $largest = $size if $size > $largest;
}

print scalar(@{$content[0]}), " line",
    (@{$content[0]} == 1 ? '' : 's'), " in original", $/;
print scalar(@{$content[1]}), " line",
    (@{$content[1]} == 1 ? '' : 's'), " in revision", $/;

print scalar(@diffs), " hunk", (@diffs == 1 ? '' : 's'),
    " differ", $/;
print $largest, " line", ($largest == 1 ? '' : 's'),
    " in largest hunk", $/;

printf "Revision %0.2f%% similar to original$/",
    100 * @LCS / @{$content[0]};
[download]

updated 2001-Aug-01: small code changes; renamed from "RE: Re: HTML Document Comparison"

Way cool! Now I can't look too hard at that if I'm going to reimplement it for the column, but way cool!

Regarding the original poster's question, can you get some quantitization of "how much" of the file is changed, like 0 to 100%?

-- Randal L. Schwartz, Perl hacker

use String::Approx 'amatch';
$match = amatch(@txt1, @txt2));
[download]

Well, better would probably be String::Similarity (which I found out only recently, and right here in the Monestary) which gives a value between 0 and 1. In fact, for the original application, that might be the easiest. {grin}

-- Randal L. Schwartz, Perl hacker

maybe with some minor modification, it looks like mdillon's code might be just what i was looking for, and seemingly far more efficient than what i proposed.

much thanks, xjar

1. Read the document and strip off everything except the html tags, including all newlines;
2. Take the MD5 hash of the tag structure;
3. Compare the MD5s of the documents to determine if they have the same structure. Or are derived from the same template with different text.

Advantadges:

* Accounts for markup changes, so differences in the text are not significant;
* If you have a lot of documents, MD5s are easy and quick to compare, as opposed to whole documents.

Disadvantadges:

* Accounts for markup changes, so differences in the text are not significant;
* MD5s are very strict, so there's no telling between small and big differences.

Actually this isn't my idea, it was used in some web survey to count the number of unique sites on the net.

Yah NetCraft. Which is a great idea, EXCEPT that the changes may be text body within a template. They were looking for templated documents specifically.

Still, a cool idea worthy of the ++ I stuck on it =)

--
$you = new YOU;
honk() if $you->love(perl)

A much more recent take on this may be found in the thread: cmp two HTML fragments

This is a little late for OP, probably, but since planetscape has cross-referenced this thread with a more recent one, what follows may still have some value for future readers

.."to determine if they are similar enough to be considered "the same"

There are some great answers, above, but note that some of them touch on what seems to me to be the threshold problem. To be explicit:

What are the criteria for sameness? For one example, are two pages "to be considered the same" if one presents a given data set as a piechart and another as a barchart? ...in tabular fashion rather than as a block of lines?

More generally, are we trying to see if:

1. the two documents render the same content with mere typographic differences occuring solely because of variant markup? (and what makes those differences "mere?")
2. the content is different, but the general appearance (layout) is the same or similar? (and for what value of "similar?")

If you sample only segments of the comparatives, how much risk of a false positive are you willing to accept?

Much as I admire the posts above, I think you need to answer these (and similar) questions for your project before adopting a method.