Re: File glob question

You haven't said which platform your program is running on. There's a good chance that it's not the machine running your code that is bottlenecked, but the file server or the network.

On Win32, there are well known performance problems with the layers of the operating system involved in wildcard directory lookup (=globbing).

I've also seen the same scenario with NFS, but nowhere near as bad - in this case, the Unix box doing the file serving had some severe hardware problems of its own.

A way round this is to use FTP instead of a direct mapping, and Net::FTP to access the directories and files.

Update: Some CB corresponence with licking9Volts has established that the files are being served from a Unix box using Samba. IIRC, Windows has to have an image of the directory in memory before it can glob. If the directory is huge, then Windows thrashes in memory.

Comment on Re: File glob question

Replies are listed 'Best First'.
Re: Re: File glob question by licking9Volts (Pilgrim) on Jun 03, 2002 at 18:23 UTC
rinceWind, I am running this on Win2k actually. The program sits in the same directory as the data files though. Would network problems still affect it? ~~Also, Abigail, would it be any faster if I gathered all of the file names into a seperate hash or array?~~ BTW, the reason, I am concerned with speed is that I would like to eventually use part of this code on an internal website as a kind of file archive search. If it really came down to it, I could just create a separate index file with the data I need, then update it weekly and search on that, but how much fun is that? Thanks. Update: I crossed out the line above because I realized that with either scenario, it's still going to glob the file names into something and therefore it still has to read ALL of the file names. Thanks Abigail.	[reply]
Re^3: File glob question by Aristotle (Chancellor) on Jun 03, 2002 at 22:11 UTC
Actually, if your CGI's being hit with any kind of traffic, you'll have to do something like that. At the very least, you'll need something from the Cache::Cache clique or one of the other similar modules. Makeshifts last the longest.	[reply]