Just a quick one regarding your side question. As far as I know there is no limit to the size of the hash its simply a question of RAM.

Ive read and dedupped a UK MPS file (4 1/2 million records) with no problem on a machine with 2 GB ram.

skywalker