There are great replies above, but note that hash collisions *could* occur even in legitimate data.

If you're worried about correctness, I suggest the following for each file:

• Normalize the data
• Compute a hash (MD5, SHA-1, whatever)
• If this is a new hash, insert $hashcode => [ $data ] into the seen dictionary.
• If this hash *does* exist in the seen dictionary, make a full compare agaisnt all datums in @{ $seen{$hashcode} }. Reject input if it matches one of the existing datums; accept and push it to the dictionary if by (unlikely) coincidence it doesn't.

I'd definitely recommend this more correct approach over the straightforward one if you need to do all this repeatedly. If you're just checking your imported data once-off, then ignoring this issue is probably fine (just keep it in mind).

But now, I've a question: don't the result pages return some sort of ID for each query result? If so, and if you can't trust this ID to guarantee data uniqueness, you should not include it in the hash. Strip anything that's not hard data — less noise, better experiment.

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.