A compromise might be to make a pass over the file once and make a DB_File database of the canonical information. Then run over the file again, and when there's missing fields consult the db for the information. That way you reduce the amount of info you have to keep in memory, at the cost of running over the file twice.