Re^2: Writing to many (>1000) files at once

It is financial data... each file is unique and even lines that have the same key may pull different items from the record, though the output is all based on the same thing.

Files can be anything from a few bytes to a couple megs, really... all depends on how many securities they ask for.

- Ant
- Some of my best work - (1 2 3)

Comment on Re^2: Writing to many (>1000) files at once

Replies are listed 'Best First'.
Re^3: Writing to many (>1000) files at once by GrandFather (Saint) on Aug 15, 2006 at 03:42 UTC
It doesn't sound like trying to "write" more than a file at a time gains anything from an I/O efficiency point of view then. You may get a gain in code organisation, but there are probably other ways to achieve that. Can you sketch the code structure and the structure of files on disk? (Not file contents, just how directories hang together and such.) How do the files get out to your customers? Is that the potential bottleneck? DWIM is Perl's answer to Gödel	[reply]

Replies are listed 'Best First'.

Re^3: Writing to many (>1000) files at once
by GrandFather (Saint) on Aug 15, 2006 at 03:42 UTC

It doesn't sound like trying to "write" more than a file at a time gains anything from an I/O efficiency point of view then. You may get a gain in code organisation, but there are probably other ways to achieve that. Can you sketch the code structure and the structure of files on disk? (Not file contents, just how directories hang together and such.)

How do the files get out to your customers? Is that the potential bottleneck?

DWIM is Perl's answer to Gödel

[reply]