Using Tie::File on such a huge file -- or any file over a few (single digit) megabytes is stupid. It will use huge amounts of cpu and be very slow.

You can build your hash much (much, much) more quickly this way:

open BIGFILE, '<', "testfile.dat" or die "Can't open file: $!\n";

my %hash;
while( <BIGFILE> ) {
    chomp;
    my( $type, $No, $date ) = split(/\|/);
    $hash{$No.$date} = $type."@".$No."@".$date;
}
close BIGFILE;

## do something with the hash.
[download]

Will use far less cpu & memory and complete in less than 1/2 the time.

However, it is really doubtful that you will be able to build a hash from that size of file without running out of memory unless:

• there are huge numbers of duplicate records in that file.
• You have a machine that has huge amounts of memory.
• You have a huge swap partition. (Preferably sited on a SSD).

Hi BrowserUK,
Thanks a lot for your inputs. I have tried your option as well, sitll the CPU usage is 100%.Is it possible to load only one line into memory from the huge file and store it into hash or without opening directly possible to store it into hash. I am worrying that it may not be very less possible. Still thought of getting your suggestion.
Thanks
madtoperl

I have tried your option as well, sitll the CPU usage is 100%.

That is because you are using more memory for the hash than you have installed, thus, some parts of the memory holding the hash are being swapped or paged to disk as the file is being read. The nature of the way hashes are stored means that pages of memory are constantly being written to disk and then re-read, over and over; and that is what is driving up your cpu usage.

Is it possible to load only one line into memory from the huge file and store it into hash or without opening directly possible to store it into hash.

That is what my code does. Read's one line installs into the hash then reads the next. It is the size of the hash that is the problem, not the line-by-line processing of the file.

I am worrying that it may not be very less possible. Still thought of getting your suggestion.

There are various ways of providing access to huge amount of data without requiring that it all be held in memory concurrently. Which of those methods/mechanisms is appropriate for your purpose depends entirely upon what you need to do with that data.

So, before advising further, you need to answer the question: Why are you attempting to load all the data into a hash?

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.
p

I'd say your best bet is to process the file one line at a time, as has already been suggested, and put the relevant information into a database. Once it's there, you can gather and extract the information you need -- and let the database engine worry about figuring out the best way to store it all.

As others have said, there are better ways to do this. But as for your question on CPU usage: the 99.9% that ps is reporting means that your perl process is using 99.9% of the CPU that is being used. So no other processes are using much to speak of. The usage reported by top is based on total capacity, so idle time is included in the total there.

To reduce a process's load on a system, see nice or the section on nice in your shell's man page.

You're saying your file is over 5 TB (terabytes) in size... is that what you actually meant? Unless your system has a few TB of memory, good luck.

The differing CPU utilization is probably from a multi-core processor. Without multi-threading, you'll never use more than 1 core at a time.

I agree with BrowserUK that loading such a huge file into memory even before you start reading the first line is a no go. You really want to iterate line by line over the file to reduce the memory used. Having said that, it is unlikely you will be able to fit 5 million megabytes into a hash.

On your CPU usage question, how many CPUs/cores do you have?

Hi Laurent R<br. Thanks a lot. I have only one CPU. Why the CPU usage says 23% in one place and 100% in other place. Is my script the CPU's whole 100% while running my script or only 23% of the whole CPU's time. Could you please clarify.
Thanks
madtoperl

Hi, my question was: how many CPUs/cores (not just CPUs) do you have. Even if you have only one (e.g. Intel) CPU, but with, say, five cores, your process might very well take more or less 100% of one core's processing power but still leave the 4 other cores almost completely idle. Since you are not forking subprocesses in your program nor using any threads, your process can basically only use one core (the system itself might be able to delegate a small fraction of its own work to another core, but this is likely to be very limited). So you might very well use 100% of one core's processing power, but only 20 or 25% of the CPU total processing power.

If you are comparing two files for common/different keys, and if both files are about the same (huge) size, I guess you will have to get smarter than keeping all the information in memory (because you don't have enough memory).

If you can make an educated guess as to where in a file a key is likely to be found, you could use seek to look for the key in the file. This is horribly slow, but likely still faster than swapping memory. If you want to be fancy, you can cache parts of the file in memory.

If you cannot make an educated guess, I guess it will pay off to convert at least one file into a file with all your keys in fixed width, sorted by the keys. Then you can easily make an educated guess to find a given key. If you convert both files to that structure, you can easily find the keys missing in one of the two files by reading through the sorted key files line by line. This approach will roughly double your disk requirements.

If your file are sorted in accordance with the comparison key, then you can iterate through the two files in parallel. This can be very very fast. Just a couple of hours ago, I compared two 100-MB files this way, it took less than 3 seconds to run.

$time perl  compare_files.pl

real    0m2.378s
user    0m1.384s
sys     0m0.069s
[download]

Even if they are not sorted, this might still be the solution: first to sort both files and then read them in parallel. The only difficulty is to get the parallel reading really correct.