Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Re: Faster Method for Gathering Data

by Cine (Friar)
on Jul 31, 2003 at 12:52 UTC ( [id://279548]=note: print w/replies, xml ) Need Help??


in reply to Faster Method for Gathering Data

Your problem is most likely not really related to perl, it is a filesystem thing, where lookups are made in linear time with regards to the number of files in a directory.
You should look into the htree option of ext{2,3}. Goggle will help you there ;)

T I M T O W T D I

Replies are listed 'Best First'.
Re: Re: Faster Method for Gathering Data
by herveus (Prior) on Jul 31, 2003 at 15:05 UTC
    Howdy!

    Your problem is most likely not really related to perl, it is a filesystem thing, where lookups are made in linear time with regards to the number of files in a directory

    I've run into what I think is similar behavior. I have some CDs that have something like 11,000 files on them, all in a single directory. On a Windows or MacOS 9 box, I saw excruciatingly slow access times for files down in the list. The first few hundred were plenty zippy, but the farther I got into the list, the slower the access.

    Doing the same access on a Solaris box or MacOSX yielded pleasantly surprising results. File lookups were more like constant time instead of proportional to how far into the list the name was.

    I suspect that the problem is exacerbated by using a "slow" medium, like CD-ROM or network volumes.

    yours,
    Michael

      Network, yes. CDROM no. The difference is that the meta data on the CDROM can be cached, whereas the network drive has to recheck it.

      T I M T O W T D I

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://279548]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others musing on the Monastery: (11)
As of 2024-04-19 16:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found