File::Find on Win32 with mapped drives

iakobski has asked for the wisdom of the Perl Monks concerning the following question:

I have a program which traverses directory trees and records the names of files that match certain criteria. This seemed like an ideal job for File::Find, and the program is fairly simple.

This program was running regularly and happily, then one day we need to run it on a load of remote drives. I mapped the drives and set off the program, it took three and a half hours to run.

A bit of testing showed that it was running at a rate of at least 1000 files a second when traversing the local drive, but this dropped to a blistering 0.5 files per second across a good connection and 0.11 files per second for a drive in the far east.

So what is going on here? I know it is slow to connect to a shared drive, but surely all it is doing is opening the directory and reading the file names? Are there any options in File::Find or any mods I can do to speed this up, or am I going to have to write my own directory traversing code instead?

-- iakobski

Comment on File::Find on Win32 with mapped drives

Replies are listed 'Best First'.
Re: File::Find on Win32 with mapped drives by Dog and Pony (Priest) on Feb 13, 2002 at 16:11 UTC
I can only assume that this would be because a new connection is initiated every time. That is, every `open` and `opendir` is connecting anew to the remote server. I am not sure what to do about this, if there is anything. Have you tried accessing files on your own (with perl, of course) and benchmarked that (so you know if it is `File::Find` or perl that has a hard time with windows shares). Also, I could think to try and toggle `no_chdir` in `File::Find` although I don't really expect any great effect from that, it is worth a shot. Maybe that affects how connections are made.	[reply] [d/l] [select]
Re: File::Find on Win32 with mapped drives by rinceWind (Monsignor) on Feb 13, 2002 at 15:06 UTC
I think that you will find that the problem lies with the way Windows scans directories and shares. Have you tried using FTP instead? I think your directory access will be much quicker with FTP. Worth trying a benchmark.	[reply]
Re: File::Find on Win32 with mapped drives by em (Scribe) on Feb 13, 2002 at 21:06 UTC
What version of Windows are you running? What version of Windows are the servers running? If you are using Win2K or NT clients/servers your performance should be decent. I've had perl scripts that process over 10K files across a local network in about 10-15 minutes. If Win9X/ME is used, you'll have a performance hit there. What kind of connection do you have to the share? You mention 'the far east', are you talking about processing files across a trans-ocean hop? I had code that copied files from a CD-ROM into a particular directory structure on a network share. The Win95 machine with the CD-ROM was in one city while the NT file server was 12 or 14 hops away in another city (about 40 miles) across several T1 lines. It took about 2 to 3 hours to copy 300MB to 400MB of data. Be default, File::Find does a CHDIR into each directory it processes. You might pick up some speed if you disabled it. i.e. find( {wanted => \&wanted(), no_chdir => 1}, 'm:/'); Finally, if the process is too slow, you might want to consider moving the data closer to you or your code closer to the data.	[reply]