in reply to performance of huge array looping

If you're dealing with that much data, do you REALLY need to load it all into memory in an array?

Given that you're only looking at each item once, it would make more sense to either process the data line-by-line from a text file (if it's that kind of data), or iterate over it record-by-record from a database using DBI.

You program will run MUCH faster, as a bonus, since you're not going to be swapping. This is definitely a case where loading things in memory won't be an optimization.
--
Mike

Replies are listed 'Best First'.
Re: Re: performance of huge array looping
by kommesel (Sexton) on Aug 15, 2002 at 08:26 UTC
    The source is an ASCII file. I've thought of it, and basically all I need is to load lines which begin with *| and work on them. and their size is much smaller than the file. So I'll read line by line, push into array only the lines that begin with *| (by the way, I need to use m/^\*\|/, don't I?) and loop over them, thus eliminating the need for swap.
[reply]
      Makes sense...

      Yes, you would need to escape those characters in your regex.
      --
      Mike

[reply]

In Section Seekers of Perl Wisdom