I have to agree with the database approach brought up earlier. I occasionally run into these issues where correlation between large files is required and have yet to be able to beat the combo of perl and DBI/DBD. SQLite has been a life-saver

• for ease of use
• quick implementation
• a short learning curve
• as well as performance savings

Best of luck in whatever approach you use...