Re: Filesystem: How to grab subset of directory structure

Your map is not nessicary and the regex could be a little more optimized.

this will be slightly faster:

aryFiles = grep /\.xml$/i, readdir MYFILEHANDLE;
[download]

But my guess is that on your FS putting 20,000 files in one directory is what is making it slow. Probably you're better off making a couple of subdirectories; about 500 files each in them might be a lot faster. - This all depends on your filesystem so YMMV.

Joost.

Comment on Re: Filesystem: How to grab subset of directory structure Download Code

Replies are listed 'Best First'.
Re: Re: Filesystem: How to grab subset of directory structure by P0w3rK!d (Pilgrim) on Jun 25, 2003 at 18:48 UTC
In this version of the application the producer is sending me the files and dumping them into one directory. Version 2.0 will use subdirectories. The problem is, that even with subdirectories, when the consumer (my code) cannot send files to a remote FTP site (server is down or whatever) the files will queue up. Hence, I am getting tens of thousands of files. Just last weekend the FTP server was down and 35,000 XML files were queued up. Even with subdirectories the problem still remains. Thank you for your reply :) -P0w3rK!d	[reply]

Replies are listed 'Best First'.

Re: Re: Filesystem: How to grab subset of directory structure
by P0w3rK!d (Pilgrim) on Jun 25, 2003 at 18:48 UTC

The problem is, that even with subdirectories, when the consumer (my code) cannot send files to a remote FTP site (server is down or whatever) the files will queue up. Hence, I am getting tens of thousands of files. Just last weekend the FTP server was down and 35,000 XML files were queued up. Even with subdirectories the problem still remains.

Thank you for your reply :)

-P0w3rK!d

[reply]