in reply to Re: Re: Line by line file compare w/ low system memory
in thread Line by line file compare w/ low system memory

This is a time when you may want to use existing commands that do this sort of thing, at least to prime your Perl code. Two that come to mind on UNIX:

  1. The comm command does what you want if the files are sorted.
  2. You could prime the process with this:
    sort file1 file2 | uniq -u
    The output of that is just the lines that one has but the other doesn't. This means you only have to read one of the files if you do this sort of thing:
    local *IN; my %diffs; open(IN, "sort file1 file2 | uniq -u |") or die "Failed to open pipe to sort/uniq command: $!\n"; ++$diffs{$_} while (<IN>); close(IN) or die "Failed to close pipe: $!\n"; open(IN, "<file1") or die "Failed to open file1: $!\n"; print "In file1 only:\n"; while (<IN>) { if ($diffs{$_}) { print $_; delete $diffs{$_}; } } print "\nIn file2 only:\n", keys %diffs, "\n";