Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options

Re: Re: Faster Method for Gathering Data

by herveus (Parson)
on Jul 31, 2003 at 15:05 UTC ( #279601=note: print w/replies, xml ) Need Help??

in reply to Re: Faster Method for Gathering Data
in thread Faster Method for Gathering Data


Your problem is most likely not really related to perl, it is a filesystem thing, where lookups are made in linear time with regards to the number of files in a directory

I've run into what I think is similar behavior. I have some CDs that have something like 11,000 files on them, all in a single directory. On a Windows or MacOS 9 box, I saw excruciatingly slow access times for files down in the list. The first few hundred were plenty zippy, but the farther I got into the list, the slower the access.

Doing the same access on a Solaris box or MacOSX yielded pleasantly surprising results. File lookups were more like constant time instead of proportional to how far into the list the name was.

I suspect that the problem is exacerbated by using a "slow" medium, like CD-ROM or network volumes.


  • Comment on Re: Re: Faster Method for Gathering Data

Replies are listed 'Best First'.
Re: Re: Re: Faster Method for Gathering Data
by Cine (Friar) on Jul 31, 2003 at 15:10 UTC
    Network, yes. CDROM no. The difference is that the meta data on the CDROM can be cached, whereas the network drive has to recheck it.

    T I M T O W T D I

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://279601]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (2)
As of 2022-10-01 04:41 GMT
Find Nodes?
    Voting Booth?
    I prefer my indexes to start at:

    Results (126 votes). Check out past polls.