in reply to How to process two files of over a million lines for changes
Unix has a nice utility called "comm" that is perfectly suited for your needs.
It takes 2 sorted input files and returns the output in 3 columns:
column 1: lines unique to input file 1
column 2: lines unique to input file 2
column 3: lines common to both files
You can read the manpage for it, but this is how you would use it:
comm -3 yesterday.file today.file > difs.txt
The -3 switch turns off the output of column 3 (which you don't need).
The file difs.txt now contains 2 columns of data: The first column is the records unique to yesterday. The second column is the records unique to today. This file can easily be parsed to separate the 2 columns.
The records in column 1 are the ones that need to be deleted from the database.
The records in column 2 are the ones that need to be added to the database.
hope this helps,
davidj