Amongst others, there is one very strong clue that your perl is limited to 4GB: ptrsize=4,. With 4-byte pointers 2**32/4GB is the maximum memory the process will be able to address. But, a good chunk of that address space will be devoted to the OS. Often the top 1GB.

So if top was showing you that 40% (3.2GB) of memory was used, with some of that occupied by other processes, system drivers etc., the realistic expectation is that your Perl programs will be limited to 3GB.

As for how to resolve the problem, the pat answer would be "use less memory", but that's probably not very helpful. Some possibilities are: build a 64-bit perl; split the data into 2 processes and have the code that uses it do so from a 3rd process via sockets or pipes; store the data more efficiently. It is often possible to reduce the memory usage, but to suggest ways you'll have to give us more information.

• Show us what a couple of the lines of that file look like.

  If they vary widely in length, then the shortest, longest and one that roughly average would be a good start.

• Show us how you are using the hash. That is, once you have created it, what do you do with it?