Making this script process 56,000 lines 5 times faster

kris004 has asked for the wisdom of the Perl Monks concerning the following question:

So, I decided to learn some Perl earlier today. I had a small script that downloaded and converted a hosts file for use with unbound:

 
#!/bin/sh
curl https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts 
+|grep '^0\.0\.0\.0' | awk '{print "local-zone: \""$2"\" redirect\nloc
+al-data: \""$2" A 0.0.0.0\""}' > ads.conf
[download]

Which discards all lines except lines like this:

0.0.0.0 apps.id.net

And changes it into two lines like this:

local-zone: "apps.id.net" redirect
local-data: "apps.id.net A 0.0.0.0"
[download]

I poked around this site and read some docs and came up with this:

#!/usr/bin/perl
use strict;
use warnings;
use LWP::Simple;

my $url = "https://raw.githubusercontent.com/StevenBlack/hosts/master/
+hosts";
open(my $file, '>', 'ads.conf') or die "wtfile?";
for (split /^/, get ($url)) {
        if ($_ =~ /0\.0\.0\.0/ and $_ !~ /#/){
                $_ =~ s/0\.0\.0\.0 //;
                $_ =~ s/\n//;
                print $file "local-zone: \"" . $_ . "\" redirect\nloca
+l-data: \"" . $_ . " A 0.0.0.0\"\n";
        }
}
close $file;
[download]

Turns out the Perl script takes 5 times as long as the shell script, and uses nearly twice the cpu. What mistakes did I make, and is there a way to do this in Perl faster then the shell script can?

Updated script based on comments:

#!/usr/bin/perl
use strict;
use warnings;
use LWP::Simple;

my $url = "https://raw.githubusercontent.com/StevenBlack/hosts/master/
+hosts";
open(my $file, '>', 'ads.conf') or die "wtfile?";
for (split /^/, get ($url)) {
        if ($_ =~ /^\Q0.0.0.0\E (.*)$/){
                print $file "local-zone: \"" . $1 . "\" redirect\nloca
+l-data: \"" . $1 . " A 0.0.0.0\"\n";  
        }
[download]

Update 2, thank you marioroy (link) and rizzo (link), combining your two suggestions gets me the fastest script so far:

#!/usr/bin/perl
use strict;
use warnings;

my $_url = "https://raw.githubusercontent.com/StevenBlack/hosts/master
+/hosts";
open(my $file, '>', 'ads.conf') or die "wtfile?";

my $var = `curl $_url` or die "wturl?";

while($var =~ /(\n0\.0\.0\.0)(\s)([\w\.]+?)\s+/g){
        print $file "local-zone: \"" . $3 . "\" redirect\nlocal-data: 
+\"" . $3 . " A 0.0.0.0\"\n";
}
close $file;
[download]

Interestingly that script, seems to be on par with the shell script time wise, but does it using 1/6th the cpu.

Comment on Making this script process 56,000 lines 5 times faster Select or Download Code

Replies are listed 'Best First'.
Re: Making this script process 56,000 lines 5 times faster by LanX (Saint) on Mar 22, 2018 at 03:25 UTC
Some guesses: You didn't anchor your regexes like in the grep example, they only need to match at the start `/^0\.0\.0\.0/` , the other regex to exclude comments is obsolet then. And use `chomp` not an (again unachored) regex to strip newlines. I might also try to use a `while (my $line = <$in> ){...}` loop instead of `for` , but you need to do something like `open my $in ,"<", \$got;` beforehand. Perl might still be slower, because it needs more time for start up and compiling. (all code untested) PS: there are more advanced tricks possible in this specific case, but you want to learn Perl and I don't wanna exaggerate it. update the substitution `$_ =~ s/0\.0\.0\.0 //;` doesn't make sense if you match the remaining of the line right away `if (/^\Q0.0.0.0\E (.)$/) { print "<<<$1>>>"; }` [download] Cheers Rolf _{(addicted to the Perl Programming Language and ☆☆☆☆ :) Wikisyntax for the Monastery} *) especially LWP::Simple! For a fair comparison try to pipe in curl.	[reply] [d/l] [select]
Re^2: Making this script process 56,000 lines 5 times faster by kris004 (Initiate) on Mar 22, 2018 at 04:08 UTC
Thanks for the tips. Disregarding the networking part, with those changes the script runs twice as fast.	[reply]
Re: Making this script process 56,000 lines 5 times faster by marioroy (Prior) on Mar 22, 2018 at 04:20 UTC
Hello kris004, Performance varies between Perl releases. It appears from testing regular expression may run slower in Perl v5.20 and higher. Therefore, provided a subsequent demonstration to factor out the regular expression engine. Update 1: Runs fast using any Perl release by merging the two regular expressions into one. Suggestion by LanX. Of course ;-), minimize the use of multiple regular expression statements when possible. Update 2: Added -w switch to Perl. `#!/bin/sh curl https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts +\|\ perl -wnl -e ' if ( /^0\.0\.0\.0 (.)$/ ) { print "local-zone: \"" . $1 . "\" redirect\nlocal-data: \"" . $1 . + " A 0.0.0.0\""; } ' > ads.conf` [download] Perl up to 5.18.x* `#!/bin/sh curl https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts +\|\ perl -wnl -e ' if ( /^0\.0\.0\.0/ ) { chomp; s/^0\.0\.0\.0 //; print "local-zone: \"" . $_ . "\" redirect\nlocal-data: \"" . $_ . + " A 0.0.0.0\""; } ' > ads.conf` [download] Perl 5.20.x and higher `#!/bin/sh curl https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts +\|\ perl -wnal -e ' if ( $F[0] eq "0.0.0.0" ) { print "local-zone: \"" . $F[1] . "\" redirect\nlocal-data: \"" . $ +F[1] . " A 0.0.0.0\""; } ' > ads.conf` [download] Perl switches `$ perl --help Usage: perl [switches] [--] [programfile] [arguments] ... -a autosplit mode with -n or -p (splits $_ into @F) -e program one line of program (several -e's allowed, omit pr +ogramfile) -l[octal] enable line ending processing, specifies line term +inator -n assume "while (<>) { ... }" loop around program -w enable many useful warnings ...` [download] Regards, Mario	[reply] [d/l] [select]
Re^2: Making this script process 56,000 lines 5 times faster by marioroy (Prior) on Mar 22, 2018 at 07:02 UTC
Hi kris004, Another option is having Perl read the Curl output directly versus using LWP::Simple. The "-\|" mode for open means the string that follows is interpreted as a command that pipes the output to us. See open function. `#!/usr/bin/perl use strict; use warnings; my $input = "https://raw.githubusercontent.com/StevenBlack/hosts/mast +er/hosts"; my $output = "ads.conf"; open my $IN, "-\|", "curl $input" or die "open error $input: $!"; open my $OUT, ">", $output or die "open error $output: $!"; while ( <$IN> ) { if ( /^0\.0\.0\.0 (.*)$/ ) { print {$OUT} "local-zone: \"" . $1 . "\" redirect\nlocal-data: \"" + . $1 . " A 0.0.0.0\"\n"; } } close $IN; close $OUT;` [download] Regards, Mario	[reply] [d/l]
Re: Making this script process 56,000 lines 5 times faster by rizzo (Curate) on Mar 22, 2018 at 07:27 UTC
To me it seems that by rebuilding grep's and awk's "line-by-line" processing you are taking Perl's drawbacks(starting the interpreter) but you don't profit from its power. Why not loop over the response directly using the /g modifier: `while($response =~ /(\n0\.0\.0\.0)(\s)([\w\.]+?)\s+/g ) { print ".... $3 ..."; }` [download]	[reply] [d/l]
Re^2: Making this script process 56,000 lines 5 times faster by kris004 (Initiate) on Mar 22, 2018 at 18:23 UTC
Hi, thank you for this tip, it makes sense as long as I'm not worried about running out of memory.	[reply]
Re^3: Making this script process 56,000 lines 5 times faster by LanX (Saint) on Mar 22, 2018 at 18:49 UTC
> as long as I'm not worried about running out of memory. Where do you see a memory problem here? Cheers Rolf _{(addicted to the Perl Programming Language and ☆☆☆☆ :) Wikisyntax for the Monastery}	[reply]
Re^4: Making this script process 56,000 lines 5 times faster by kris004 (Initiate) on Mar 22, 2018 at 18:53 UTC
Re^5: Making this script process 56,000 lines 5 times faster by LanX (Saint) on Mar 22, 2018 at 18:58 UTC
Some notes below your chosen depth have not been shown here
A reply falls below the community's threshold of quality. You may see it by logging in.

update