in reply to Perl with a Samba mount

When you say: When I run this script over 60,000 files on my local machine it takes about 6 minutes. When I run it over the files on the Samba mount it takes over two hours. What are you doing to these files? are you just looking at file names? are you stating multiple times for size mod date etc? are you slurping the data? Show us some code. Off the top of my head, I can think of a few things that can go wrong here, for instance grabbing multiple items such as mod time size etc can make multiple requests over the network for each file unless you do it correctly If you are slurping files to do proccessing of them *welp* the network is going to be way slower than a local hard disk. Give me more info on what you are doing and I can give you some pointers.

-Waswas

Replies are listed 'Best First'.
Re: Re: Perl with a Samba mount
by fletcher_the_dog (Friar) on Jan 04, 2003 at 00:05 UTC
    I am slurping the files in, rearranging some stuff and then writing them back out. Sorry I can't really give more details than that. Through some experimentation I have found that writing thing back out is where the big slow down is. I have found that if I "buffer" my write outs by writing out a bunch of files at a time it speeds things up ~5 percent. I know that it is the network that is slowing things down. I am wondering if there are any work arounds to make it faster over the network or make my read and writes over the network in bigger blocks so I don't have make as many network calls.
[reply]
      in short:

      read and write in 32k or 64k blocks (to try to get SMB to go into burst mode).
      stat all needed stat vars in one call.

      Even with this your bottleneck is the network -- CIFS is not the fastest network protocol under the sun...

      -Waswas
[reply]

In Section Seekers of Perl Wisdom