in reply to Alternate for "open"
Hey there, thanks for posting that. You still haven't posted the relevant code portion, so we're still guessing here.
(If your code is "too complicated" for you to be able to copy and paste 10 or 20 lines that contain the functionality you suspect for the memory hogging, then that's an indication that you should simplify the structure of the code).
I'm not a sysadmin but it seems to me that if your 64GB RAM server is running at 75% memory usage before you start, it's working rather hard. (On the other hand 25% of 64Gb is still a lot and you should be able to process any number of text files if you code it right.) Currently my busiest server which runs three different daemons forking processes, scraping websites, loading and processing data, and pushing the data to external apis, is only using about 6GB RAM.
So looking at the rest of the data you posted, it seems more likely than ever that you need a database for your data. Basically your hash is a database, but it's not up to the task.
If you have 3800 files, and if you need all the data from all the files to accomplish your task, and if reading and temporarily processing a file can add 4Mb RAM usage, you really need to use a database.
You probably already have a RDBMS on the server, but I'd start with SQLite anyway for its light memory footprint and ease of use.
It's very unlikely that you need to hold all the data in memory to do your work, and there are many good reasons why you shouldn't. For example, when you create a hash (or any data structure) in Perl, that memory is not released until the variable goes out of scope. So if you are creating a global hash to store the file data, and then reading from the hash once (for example, to add to an aggregate hash), and then not using the hash again, you're not getting that memory back before the program finishes. This can cause you to run out of memory quickly. So if your program looks anything like:
... then you are allocating memory to the hashes before you need to (though they will be empty) and keeping the memory allocated long after the usefulness of the hash has expired.#!/usr/bin/perl use strict; use warnings; my %h1; my %h2; #etc # a bunch of code here to setup and prepare, # maybe find the list of files foreach my $file ( @files ) { # open ... # split ... $h1{ $file } = $splitted[1]; $h2{ $file } = $splitted[2]; # ... and so on my $res1 = my_func( %h1 ); # don't do this my $res2 = my_func( %h2 ); # ... if ( some condition ) { # do something with $res1 and $res2 } # continue, # lots of code # working hard # but never using the hashes again }
So declare and use your variables in the smallest possible scope.
Also don't pass around actual data structures (as I do in the above example, to show bad practise), because then Perl has to make another copy of the data. Pass references to them, ie use:
my $res = my_func( \%hash ); # not: my_func( %hash );
To take that even further, consider off-loading the work of processing the files to another program, so that all system resources are released when the processing is done.
If the above tips seem random and maybe unrelated, it's because you haven't shared the code, so I'm just throwing out random and maybe unrelated tips.
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^2: Alternate for "open"
by ravi45722 (Pilgrim) on Nov 18, 2015 at 04:24 UTC |