The villain here is probably the %main_data hash, which is filling up with all your fetched data. So it's not when you get it from the DB, it's when you squirrel it away for use in your program.

The minimal mutilation solution is to do what OS's do to extend their memory: use the hard drive. You can use the MLDBM module which allows you to store complex Perl data structures on disk and manipulate them transparently (when you tie the data structure).

Not completely related to that suggestion: something that might help you make this stretch of code more maintainable: use a hash slice and an array slice instead of those multiple assignments:

@fields = qw(Resd Edlv); # just a sample
@main_data{curpidm}{@fields} = @data[1,2];
[download]

Update gotta check that, not sure it will work as advertised ...

Second update it won't ... the handy @hashname shorthand is short-circuited by the fact that we've got a reference here. So, unless somebody else figures out the syntax, disregard this suggestion.

HTH

Philosophy can be made out of anything. Or less -- Jerry A. Fodor