in reply to Perl version of 'diff'?

If you move to a real DB (or even with a slight modification to the current format) there is a useful trick - add an update time field. Unix epoch time is fine. Thus your master database looks like:

data_field1 data_field2 ... update_time

With the update time to hand it is a simple matter to select all the records with an update time > some_value and thus generate a record set than encapsulates all the latest changes, valid from any fixed time point.

It looks to me as though your data storage format is going to give you issues but this depends on what you are actually doing. There are essentially 3 types of data changes you may need to deal with:

  1. INSERTS - adding a brand new entry
  2. DELETIONS - removing dross
  3. UPDATES - ie what if username@domain.com changes their username? You either need to UPDATE their record or delete the old record and insert the new one.

cheers

tachyon