Comapring files to see if one file's records is in another file or not.

jatinpatel has asked for the wisdom of the Perl Monks concerning the following question:

Hello, I have two large files. They both have common key field, but in different place. I want to see if file_1's records exist in file_2 or not. If exists then I would like to know which one. Is there any readily available routine in perl or not. If not what is the best way to do this. Both files are very large.

Comment on Comapring files to see if one file's records is in another file or not.

Replies are listed 'Best First'.
Re: Comapring files to see if one file's records is in another file or not. by Corion (Patriarch) on Jul 19, 2007 at 19:06 UTC
What you want is explained in the perlfaq4, How do I compute the intersection of two arrays. Also, I recently wrote a program that might be applicable to your problem directly, join - join two files according to a common key. The easy solution to both problems is to basically load one of the two files (likely, the smaller one) into memory and check every entry of the second file against the first.	[reply]
Re: Comapring files to see if one file's records is in another file or not. by CountZero (Bishop) on Jul 19, 2007 at 20:28 UTC
List::Compare allows you to do a lot of different types of comparisons (intersection; union; unique; ...) on lists. I'm sure that with a bit of coaxing you can get your files into a format that List::Compare can digest. Its one drawback is that all the data must fit into memory, so that rules out working with really big lists. CountZero A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James	[reply]