in reply to How to process two files of over a million lines for changes
I know nothing of databases and little about Perl, but would it work out to have two files, one live and one for updating? When an update is done, either swap names with the live file or toggle some reference pointer to which one is 'live' and which is for 'updating'.