Re^3: File Size limits

I've been crafting workarounds which open new files every time you approach 2GB, but that's getting tiresome.

You may want to look at my module File::LinearRaid which lets you access multiple (sequential) files seamlessly using a single filehandle. It was conceived to help seamlessly overcome OS filesize limiations (among other things).

One of the ideas I had with F::LR was that you could have an enormous logical file split into reasonably-sized physical files and use BigInts as (logical) seek offsets. Since the underlying (physical) seeks would still be "reasonably" sized, it should work.. in theory! Unfortunately, I'm still stumped as to how to test this out. In fact, what I just outlined may even work in the module's current state -- I just don't know.

Also, right now there is no mechanism to automatically grow the logical file, although there is a manual mechanism to append physical files to the big logical file.

Anyway, if you think this module could work for you, let me know. I'd be happy to hear your feedback and suggestions.

blokhead

Comment on Re^3: File Size limits

Replies are listed 'Best First'.
Re^4: File Size limits by creamygoodness (Curate) on Aug 01, 2005 at 17:44 UTC
I'd actually checked out your module already. :) It looks like a nifty solution to a vexing problem. For Kinosearch, I'm content to have systems which can't deal with large files run up against the file size limit. If you're running a search engine where the index exceeds 2 Gb, you're running a serious search engine, and you're probably not running it on a machine that can't deal with >2 Gb files. My main reason for writing the workarounds was a perhaps mistaken impression as to what maximum file size 32-bit Perl's internal file manipulation routines can handle safely. WRT to problem of auto-growing a file, I was doing it by using a scalar write buffer. That stopped me from having to check if I was exceeding the 2Gb limit every single call to print(). Perhaps a similar solution could work for your module? -- Marvin Humphrey	[reply]

Replies are listed 'Best First'.

Re^4: File Size limits
by creamygoodness (Curate) on Aug 01, 2005 at 17:44 UTC

For Kinosearch, I'm content to have systems which can't deal with large files run up against the file size limit. If you're running a search engine where the index exceeds 2 Gb, you're running a serious search engine, and you're probably not running it on a machine that can't deal with >2 Gb files. My main reason for writing the workarounds was a perhaps mistaken impression as to what maximum file size 32-bit Perl's internal file manipulation routines can handle safely.

WRT to problem of auto-growing a file, I was doing it by using a scalar write buffer. That stopped me from having to check if I was exceeding the 2Gb limit every single call to print(). Perhaps a similar solution could work for your module?

-- Marvin Humphrey

[reply]