FrankG is right on target. I would recommend Berkeley DB, either through DB_File or BerkeleyDB. I believe that it will be substantially more efficient memorywise than native Perl data structures. So you may be able to avoid using a temp file by using an in memory database.

Alternately if you have a database, depending on the types of manipulations you are doing it may be easier and faster to access it with DBI, create a scratch table, populate that, and then pull data back out rather than writing your own processing logic.

A few other notes. First of all judging from your code you are still using C-style loops. Code that uses C-style loops will (like C) have various indexing errors (off by one, fencepost) as among your most common bugs. If you switch to using foreach-type loops you can eliminate that. Also I find it very helpful to use strict religiously which you don't appear to be doing. Sure it is a (small) pain to type my everywhere. But it catches so many typos that it pays off in spades...