Re: Re: reading (caching?) large files

Sorry if I'm not following, but:

1. The filesystem caches into its swap space whenever it reads into memory something too large. Do you mean it also reads ahead when Perl opens a handle, or has read a certain amount, so that it saves time?

2. No time to use anything like BerkeleyDB now, but I'll remember it for the future, though, it sounds useful.

3. What exactly do you mean row-read-write? Regular line by line? How is that helpful?

10x again,
me

--- Find the River

Edit by tye

Comment on Re: Re: reading (caching?) large files

Replies are listed 'Best First'.
Re:{3} reading (caching?) large files by jeroenes (Priest) on Jun 05, 2001 at 17:25 UTC
I was too brief, apparently. The filesystem caches pages, not files. So while perl is reading line by line, it probably reads often from the same page. Every time that page is read from cache. Works quite efficient for sequential reads. The DB is quite easy to use, it has a tied interface (aka you can approach it just like a hash). Indeed regular line by line. That way you can reduce the size of the data to reduce memory usage. For example, you can remove double spaces, remove unneeded data, or write numbers as bytes, etc etc, without having to store everything in memory. Jeroen "We are not alone"(FZ)	[reply]

Replies are listed 'Best First'.

Re:{3} reading (caching?) large files
by jeroenes (Priest) on Jun 05, 2001 at 17:25 UTC

The filesystem caches pages, not files. So while perl is reading line by line, it probably reads often from the same page. Every time that page is read from cache. Works quite efficient for sequential reads.
The DB is quite easy to use, it has a tied interface (aka you can approach it just like a hash).
Indeed regular line by line. That way you can reduce the size of the data to reduce memory usage. For example, you can remove double spaces, remove unneeded data, or write numbers as bytes, etc etc, without having to store everything in memory.

Jeroen
"We are not alone"(FZ)

[reply]