EDIT: Suggestions by jwkrahn have been included in the script


This is a baby perl version of BoneKracker bash scripts for creating ipsets based on internet sources. I have been happily using those scripts and, as I am trying to learn some Perl, I have made a perl version of them. This is the outcome.

#!/usr/bin/env perl # Perl version of BoneKracker scripts for dynamic list of network bloc +ks # (see: http://forums.gentoo.org/viewtopic-t-863121-start-0.html) # # It: # - Downloads from these sites the list of networks that should be blo +cked # http://feeds.dshield.org/block.txt # http://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt # http://www.ipdeny.com/ipblocks/data/countries/$_.zone # - checks if the downloaded file is newer than what we have # - defines the ipset # - sets a list of ipset lists, to be able to block all the networks w +ith one iptables command, # like, for example, "iptables -A INPUT -i -lo -p all -m set --match +-set ipdeny src -j DROP" # {{{ packages used use Net::Ping; use warnings; use strict; use feature qw(say); use FileHandle; use LWP::UserAgent; # }}} # Basic Variable definition {{{ # Country codes can be obtained from: http://www.ipdeny.com/ipblocks/ my @countries = qw(cn vn); my $urls_number = 2 + @countries; # dshield + bogons + numb +er of countries my (@dates_last, @dates_now, @sys); # We compare stored and + present dates my $f_dates_last = "/root/data/ipset_dates_last.txt"; # File where +dates are stored # }}} # Check if we have connection (ping google.com) {{{ my $p = Net::Ping->new("icmp"); # $p->ping("google.com") == 1 or die "Unable to ping google.com"; $p->close(); # }}} # Get stored dates {{{ my $fh = FileHandle->new; if ($fh->open("< $f_dates_last")) { @dates_last = map /^(\d+)/, $fh->getlines; $fh->close; } else { say "Can't open file $f_dates_last. Assuming this is the first tim +e we run the script and setting dates to 0"; @dates_last = ( 0 ) x $urls_number; } # }}} # Define urls to fetch and regex to match {{{ my $i = 0; my @urls = ( # url, regex, stored date, set_type, set_name [ "http://feeds.dshield.org/block.txt", # Must match: qr/^((?:\d{1,3}\.){3}\d{1,3})/m, # 46.137.194.0 46.137.1 +94.255 24 2650 $dates_last[$i++], # ^^^^^^^^^^^^ "hash:ip --netmask 24 --hashsize 64", # all class C network +s "block" ], [ "http://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt +", qr/^(\d.*\d)/m, # 14.102.160.0/19 $dates_last[$i++], # ^^^^^^^^^^^^^^^ "hash:net", "bogons" ] ); for (@countries) { push @urls, [ "http://www.ipdeny.com/ipblocks/data/countries/$_.zone", qr/^(\d.*\d)/m, # 1.0.1.0/24 $dates_last[$i++], # ^^^^^^^^^^ "hash:net", $_ ]; } # }}} # Create ipdeny ipset (storing all other ipsets) and flush {{{ @sys = qw(ipset create -exist ipdeny list:set); system(@sys) == 0 or die "Unable to create ipdeny global set because: +$?"; @sys = (qw(ipset flush ipdeny)); system(@sys) == 0 or die "Unable to flush ipdeny global set because: $ +?"; # }}} # Create sets from the defined data, and return date if new networks f +ound {{{ foreach (@urls) { # Pass ArrayRef to sub create_ipset my ($date_now,$sucess) = &create_ipset(@{$_}); push @dates_now, $date_now; my $i = pop(@{$_}); if ($sucess) { @sys = (qw(ipset add ipdeny), $i); # Add defined sets in ip +deny set list system(@sys) == 0 or die "Unable to add $i to global ipdeny se +t because: $?"; } else { print "Unable to add $i to global ipdeny set because page did +not respond \n\n"; } } # }}} # Store @dates_now in file, adding the end of line (\n) {{{ $fh->open("> $f_dates_last") || die "Unable to save timestamp urls in +$f_dates_last: $!"; print $fh map "$_\n", @dates_now; $fh->close; # }}} # Save ipset {{{ @sys = qw(/etc/init.d/ipset save); system(@sys) == 0 or die "Unable to save ipsets because $?"; # }}} # SUB create_ipset: get date_now and create ipset if newer than date_l +ast {{{ sub create_ipset { my ($url,$regex,$date_last,$set_type,$set_name) = @_; my $request = HTTP::Request->new(GET => $url); my $ua = LWP::UserAgent->new; my $response = $ua->request($request); # check if we can get an answer from the url {{{ if ($response->is_success) { my $date_now = $response->last_modified; if ($date_now > $date_last) { my @sys = (qw(ipset create), "temp_$set_name", split ' ',$ +set_type); system(@sys) == 0 or die "Unable to create temp_$set_name +of type $set_type, because: $?"; my $resp = $response->content; while ( $resp =~ /$regex/g ) { # Set the ipset while th +ere is a regex match @sys = (qw(ipset add), "temp_$set_name", $1); system(@sys) == 0 or die "Unable to add $1 to temp_$se +t_name, because: $?"; } @sys = (qw(ipset create -exist), $set_name, split ' ',$set +_type); system(@sys) == 0 or die "Unable to create $set_name of ty +pe $set_type, because: $?"; @sys = (qw(ipset swap), "temp_$set_name", $set_name); system(@sys) == 0 or die "Unable to swap temp_$set_name wi +th $set_name, because: $?"; @sys = (qw(ipset destroy), "temp_$set_name"); system(@sys) == 0 or die "Unable to destroy temp_$set_name +, because: $?"; my $cron_notice = "IPSet: $set_name updated (as of: $date_ +now)."; @sys = (qw(logger -p cron.notice), $cron_notice); system(@sys) == 0 or die "Unable to send: $cron_notice to +logger, because: $?"; } return ($date_now,1); } else { print "Unable to open $url \n "; return ($date_last,0); } # }}} } # }}}

Replies are listed 'Best First'.
Re: Create a ipset for blocking networks based on internet sources
by jwkrahn (Abbot) on Apr 23, 2012 at 05:15 UTC
    my $content = $response->content; my @network = split(/\n/,$content); @network = map { m/$regex/ ? $1 : ()} @network; ... foreach (@network) { system("ipset add temp_$set_name $_"); }

    You don't do any error checking on the contents of @network so why not just:

    ... foreach ( $response->content =~ /$regex/g ) { system("ipset add temp_$set_name $_"); }


    system("ipset create temp_$set_name $set_type"); foreach (@network) { system("ipset add temp_$set_name $_"); } system("ipset create -exist $set_name $set_type"); system("ipset swap temp_$set_name $set_name"); system("ipset destroy temp_$set_name"); my $cron_notice = "IPSet: $set_name updated (as of: $d +ate_now)."; system("logger", "-p", "cron.notice", $cron_notice);

    You don't do any error checking on the system calls.    Perhaps something like this:

    0 == system 'ipset', 'create', "temp_$set_name", $set_ +type or warn "Cannot execute ipset because: $?";


    @dates_last = $fh->getlines; @dates_last = map(m/^(\d+).*$/,@dates_last);

    You don't really need two steps there and you don't really need .*$ at the end of the regular expression:

    @dates_last = map /^(\d+)/, $fh->getlines;


    @dates_last = split(/ /,"0 " x $urls_number)

    Or just:

    @dates_last = ( 0 ) x $urls_number


    '(^([0-9]{1,3}\.){3}[0-9]{1,3}).*$', ... '^(\d.*\d).*$', ... '(.*)',

    You should compile your regular expressions here using the qr operator and the .*$ at the end is superfluous.

    qr/(^([0-9]{1,3}\.){3}[0-9]{1,3})/, ... qr/^(\d.*\d)/, ... qr/(.*)/,


    @dates_now = map {$_ . "\n"} @dates_now; print $fh @dates_now;

    No need for two steps:

    print $fh map "$_\n", @dates_now;

      Jwkrahn, thanks very much for the comments and the links where to find the information. It has been great to have some insight on how to reduce code and to understand the logic behind it. It has also been useful to look at the system command and see how it works with lists (so I now use lists in all the system calls instead of scalars). I have spend some time getting the meaning of ( 0 ) x $ulr_number. Good trick!

      I have included your suggestions into the original message. Hope this is all right.

      Very impressed with perl expressiveness: easy to start write code, tricky to master it! For somebody like me without computer knowledge or experience is very exciting! Very greateful!

      Cheers!

        You're welcome.

        I made a little mistake in one sugestion:

        qr/(^([0-9]{1,3}\.){3}[0-9]{1,3})/, ... qr/^(\d.*\d)/, ... qr/(.*)/, ... foreach ( $response->content =~ /$regex/g ) {

        And you "corrected" thusly:

        qr/\n(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/, # 46.137.194.0 + 46.137.194.255 24 2650

        The problem with that is that that pattern will match every line except the first line.    The proper solution is to use the /m option so that the pattern will match at the beginning of every line:

        qr/(^([0-9]{1,3}\.){3}[0-9]{1,3})/m, ... qr/^(\d.*\d)/m, ... qr/(.*)/, ... foreach ( $response->content =~ /$regex/g ) {


        my @sys = (qw(ipset create), "temp_$set_name", split / /,$set_ +type); ... @sys = (qw(ipset create -exist), $set_name, split / /,$set_typ +e);

        The use of / / with split may not do what you want, and it certainly is not what the shell would do.    You should use ' ' instead:

        my @sys = (qw(ipset create), "temp_$set_name", split ' ',$set_ +type); ... @sys = (qw(ipset create -exist), $set_name, split ' ',$set_typ +e);


        $fh->open("> $f_dates_last") || die "Unable to save timestamp urls in +$f_dates_last: $?";

        The $? variable will have no useful information if open fails.    You should use $! or $^E instead.