in reply to Script far too slow with large files - wisdom needed!
If this is not an one-off exercise use a database system.