Re: BASH vs Perl performance

1-wget approx. 5000 rar files total from 2 different servers
Depending on what you are doing here with wget (FTP, HTTP, HTTPS) you can use LWP or Net::FTP to avoid running 5000 instances (exec's) of wget.
2-unrar the archives into approx. 16000 files of 4 different types
This can't really be made faster with perl.
3-move/rename the files into different directories based on file type
Avoid this step altogether (do the final placement while doing step 5).
4-run one group of files through a series of sed filters producing a modified set of files
You can run this through your script without invoking sed so many times. Perl has built in functionality that can do this task on one process.
5-copy all files (now approx. 20000) into a matching directory structure on a Windows box via smbclient
Copy (to the renamed new directory target) the files to the structure that you need o the Windows server at this time. Do not need to move stuff around in step 3 at all.

-Waswas

Comment on Re: BASH vs Perl performance

Replies are listed 'Best First'.
Re^2: BASH vs Perl performance by jcoxen (Deacon) on Aug 11, 2004 at 16:10 UTC
Re: your points... 1. It's pointed out elsewhere but I'm only doing 2 wgets, not 5000 3&5. I thought about combining these when I was writing the original script but to do that, I'd have to fire off smbclient several thousand times. Given my past experience with Windows networking, this struck me as something to be avoided at all costs. :) 4. I don't doubt this at all. This is a ver 1.0 script so I haven't done anything about tightening up the code yet. I've been concentrating on getting it fully functional and running without any problems. It was 3 (and to a lesser extent, 4) that prompted my question about porting to Perl. After reading everyone's responses, I think I'll go ahead and port it over. Worst case is I learn some new stuff. Best case is the process runs a whole lot faster. Thanks for your comments, Jack	[reply]

Replies are listed 'Best First'.

Re^2: BASH vs Perl performance
by jcoxen (Deacon) on Aug 11, 2004 at 16:10 UTC

1. It's pointed out elsewhere but I'm only doing 2 wgets, not 5000
3&5. I thought about combining these when I was writing the original script but to do that, I'd have to fire off smbclient several thousand times. Given my past experience with Windows networking, this struck me as something to be avoided at all costs. :)
4. I don't doubt this at all. This is a ver 1.0 script so I haven't done anything about tightening up the code yet. I've been concentrating on getting it fully functional and running without any problems.

It was 3 (and to a lesser extent, 4) that prompted my question about porting to Perl. After reading everyone's responses, I think I'll go ahead and port it over. Worst case is I learn some new stuff. Best case is the process runs a whole lot faster.

Thanks for your comments,

Jack

[reply]