in reply to Compare two Large FlatFiles

From the description you gave this psuedo code may match the problem (yeah, I'm being hesitant)

if ( file1-field2 eq file2-field5 ) { give me file2-field_to_be_named || first_round_draft_pick }
OK, kidding about the draft pick.

If this is close to the truth, consider these fragments to work with:

my %file1; my %file2; while(<file1>) { my $field_from_1 = (split /<DELIMiTER>/)[1]; $file1{$field_from_1}++; } while(<file2>) { my ($field_from_2, $record_we_want) = (split /<DELIMITER>/)[4,index +_we_cant_guess_from_description]; $file2{$field_from_2} = $record_we_want; } foreach my $keys ( %file2 ) { if ( exists $file1{$key} ) { # do whatever it is they want with $record_we_want } }
Those snips require a reasonably small subset of the memory needed to do what you described. Perhaps small enough to do the entire task in memory. What they don't do is: If this is not helpful I'll join the first two responders in requesting you provide more information. A few lines of sanatized data file and a sample of the code you've developed so far will do wonders for our helpfulness.

Update: destressed some tortured English


Be Appropriate && Follow Your Curiosity

Replies are listed 'Best First'.
Re^2: Compare two Large FlatFiles
by suneel.reddy (Novice) on Apr 17, 2012 at 09:25 UTC
    Hi guys, below is my piece of code....please help me in tuning this...only logic I can think with the knowledge I have on Perl :) while(my $pline = <PRFILE>) { $parrecord = $pline; my @parfields=split('\|',$parrecord); chomp(@parfields); # Child file is already loaded into @carray foreach (@carray) { $chrecord = $_; @chfields=split('\|',$chrecord); chomp(@chfields); if(@chfields4 eq @parfields1) { # Some logic # last ; } } } OOOH God...!!!! I was shocked to see the alignment of my code after posting this, but I have no option :)
      Okay here is my problem.....

      record from file1 :

      I|1400000042597061|ACTV|602282|2011-08-29||602178|JUSTIN||MAGRUDER||||||602282|100001|||||Gold||600990|||||||WUSA00029582|529381||||||||||||

      record from file2 :

       I|1400000042589325|2011-08-29|ACTV|1400000042597061|600002|||1556 3RD AVE|||NEW YORK|NY|10128|3100|US|||||||

      Here second field from F1 is eual to 5th field of F2

      And my script...

      while(my $pline = <PRFILE>) { $parrecord = $pline; my @parfields=split('\|',$parrecord); chomp(@parfields); # Child file is already loaded into @carray foreach (@carray) { $chrecord = $_; @chfields=split('\|',$chrecord); chomp(@chfields); if(@chfields[4] eq @parfields[1]) { # Push $chrecord into some array# last ; } } }