in reply to Quickest way to write multiple files

Copied from chatterbox: saskaqueer's mind boggles over the newest SoPW. Imagine a "magical module out there" that could "save a bunch of files with only one I/O call". That'd be amazing to see ;)

That aside, I'd say 20 seconds isn't too bad for creating and manipulating 417 files. If your code can use any adjustments for speed, it would most likely be with the code you are using to split up a chunk of data into 12 records per file. If the bottleneck here is indeed the file I/O for outputting the new files, then that's about the best you're going to get.

How are you reading in this 5000+ record file? Are you reading it in all at once into memory, or are you looping through it 12 records at a time to split it up? If you're pulling everything into memory at once, that might be your problem there.

  • Comment on Re: Quickest way to write multiple files

Replies are listed 'Best First'.
Re^2: Quickest way to write multiple files
by Anonymous Monk on Jun 08, 2004 at 08:48 UTC
    The parsing of the initial datafile is taking an average of 2 seconds. Of couse I'll try to shave 1.5 off, but that's not much. They are being stored in a hashref, then split into separate hashrefs, and output as xml 12 at a time with XML::Simple.

    The actualy splitting of the records is only taking around .1 seconds, so not much I can do there.

    Thanks anyway. If I happen to stumble on a way to make the magic module I'll be sure to post it here.
[reply]

In Section Seekers of Perl Wisdom