Increase speed script

marto9 has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I'm a beginner in perl and I made some code to clean log files. But the speed isn't fast enough. So I was wondering if a more advanced user could change it a little. Thx in advance

#!/usr/bin/perl
use warnings;
use strict;
use Term::ReadKey;
print "[-] Enter the filename: ";
chomp($file = <STDIN>);
open(LIST,$file);
@file1 = <LIST>;
close(LIST);
if (-e "cleaned.txt") {
    print "[-] Can cleaned.txt be deleted? ";
    $answer = <stdin>;
    if ($answer =~ /yes/i) {
        unlink "cleaned.txt";
    }
    else {
        &exit
    }
}
print "[-] Cleaning...\n";
foreach $data (@file1) {
    ($w1) = split(/:/,$data);
    open(OUTPUT,">>cleaned.txt");
    print OUTPUT $w1."\n" unless ($data =~ /-/);
    close(OUTPUT);
}
&exit;
sub exit() {
    ReadMode 2;
  print "\n[-] Press ENTER to exit...";
  <STDIN>;
  exit;
}
[download]

edit: Sorry people. I just saw that I posted this in the wrong section. Can a mod please move this.

Comment on Increase speed script Download Code

Replies are listed 'Best First'.
Re: Increase speed script by roboticus (Chancellor) on Jul 11, 2008 at 13:21 UTC
marto9: I believe that the primary speed problem is due to you opening and closing the output file in your cleaning loop. You should open the file before starting the loop, then write out all your data, and close at the end: `print "[-] Cleaning...\n"; open(OUTPUT,">>cleaned.txt") or die "Can't open cleaned.txt!"; foreach $data (@file1) { ($w1) = split(/:/,$data); print OUTPUT $w1."\n" unless ($data =~ /-/); } close(OUTPUT) or die "close error";` [download] A couple of minor items: Always check your file operations (the `or die "msg"` bits above) to ensure that they were successful. If cleaned.txt is a "valuable" file, you should probably delete it only after successfully building your new one. You could, for example, build the file as `cleaned.txt.tmp`. Then, only after the final close is successful, you could rename `cleaned.txt` to `cleaned.txt.bak` and then rename `cleaned.txt.tmp` to `cleaned.txt`. ...roboticus UPDATE: Fixed a bit of awkward grammar, added some formatting for readability.	[reply] [d/l] [select]
Re^2: Increase speed script by marto9 (Beadle) on Jul 11, 2008 at 14:27 UTC
Thx roboticus! That speeded up my script A LOT. :) And I also added some "or die" code in it.	[reply]
Re: Increase speed script by karavelov (Monk) on Jul 11, 2008 at 16:57 UTC
It the files to be cleaned are quite big it is better to process them line by line. I have in mind something like `open LIST,$file; open OUTPUT,">>",'cleaned.txt'; print "[-] Cleaning...\n"; while (<LIST>) { ($w1) = split(/:/,$_); print OUTPUT $w1."\n" unless ($data =~ /-/); } close LIST; close OUTPUT;` [download]	[reply] [d/l]
Re^2: Increase speed script by marto9 (Beadle) on Jul 11, 2008 at 18:07 UTC
Ty for you advice. But could you pls tell me why it's better to use the while loop instead of the for loop?	[reply]
Re^3: Increase speed script by massa (Hermit) on Jul 11, 2008 at 19:48 UTC
Because the for brings the whole file to the memory before walking thru the lines. This is usually slower than going line-by-line in the file (as the while does) because: it fills many pages of the memory with the contents of the file, potentially swapping stuff out -- instead of just allocating a couple of pages for a file buffer; then it walks thru those pages, potentially using cache lines -- instead of pulling the file buffer to the same cache line over and over, freeing cache lines to other stuff; goes to the disk all at once, blocking until it has read everything -- instead of going to the disk when it had already processed the last chunk of information and then blocking for less time. []s, HTH, Massa	[reply]
Re: Increase speed script by kyle (Abbot) on Jul 11, 2008 at 17:06 UTC
The suggestion from roboticus is a good one. I'd like to add to that a more general suggestion. When you have a performance problem, there are tools available to help tell where the problem is. Using Devel::DProf or Devel::Profile, you can get profiling data on a per-sub basis. For a tutorial, see Profiling your code. For something like what you have, where there aren't many subs, just a short program of lines, Devel::NYTProf and Devel::SmallProf are more appropriate tools. Try them out. You may be surprised at what they show (I usually am).	[reply]