Need a better way to break out a range of addresses...

RMaxwell has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Need a better way to break out a range of addresses... by ikegami (Patriarch) on Mar 28, 2007 at 20:43 UTC
Using Algorithm::Loops::NestedLoop: `use Algorithm::Loops qw( NestedLoops ); sub expand { my $r = (@_ ? $_[0] : $_); my @ends = (split(/-/, $r), $r); return [ $ends[0] .. $ends[1] ]; } while (<DATA>) { s/#.//; s/\s+\z//; my @ranges = map expand, split /\./, $_; my @ips = NestedLoops( \@ranges, sub { join('.', @_) }, ); foreach my $ip (@ips) { print("$ip\n"); } } __DATA__ 172.17.119.2 # Comments are here... 172.17.119.4-5 # Comments are here... 172.19-21.254.2-3 # Comments are here... 192.168.1.1-3 # Comments are here...` [download] The above can also be written as an iterator: `while (<DATA>) { s/#.//; s/\s+\z//; my @ranges = map expand, split /\./, $_; my $iter = NestedLoops(\@ranges); while (my @parts = $iter->()) { my $ip = join('.', @parts); print("$ip\n"); } }` [download] Update: Factored out `expand`. It makes it much easier to read.	[reply] [d/l] [select]
Re: Need a better way to break out a range of addresses... by grinder (Bishop) on Mar 28, 2007 at 20:02 UTC
At first I thought something like Net::IP or NetAddr::IP could handle this, but apparently they don't. I would just write some regexps to pull this apart. The following is pretty fragile in that it will fail to deal with bogus data like `10.11.12.30-29`, but that should be easy enough to trap. `use strict; use warnings; while (<>) { chomp; s/\s#.$//; if (my ($stem, $first, $last) = /\b(\d+(?:\.\d+){2})\.(\d+)-(\d+)\ +b/) { print "$stem.$_\n" for $first..$last; } elsif (/\b(\d+(?:\.\d+){3})\b/) { print "$1\n"; } }` [download] update: gah. I didn't initially spot that other octets apart from the last octet might have ranges in them. That means this solution is too simple-minded to be of much use. • another intruder with the mooring in the heart of the Perl	[reply] [d/l]
Re: Need a better way to break out a range of addresses... by Anno (Deacon) on Mar 28, 2007 at 20:34 UTC
Here is one way of doing it: `my $re = join '\\.' => ( '([-\\d]+)' ) x 4; while ( <DATA> ) { my ( $p0, $p1, $p2, $p3) = map [ expand( $_)], /$re/; for my $i0 ( @$p0 ) { for my $i1 ( @$p1 ) { for my $i2 ( @$p2 ) { for my $i3 ( @$p3 ) { print "$i0.$i1.$i2.$i3\n"; } } } } } sub expand { my ( $from, $to) = split /-/, "$_[ 0]-$_[ 0]"; $from .. $to; }` [download] Anno	[reply] [d/l]
Re: Need a better way to break out a range of addresses... by ptum (Priest) on Mar 28, 2007 at 19:42 UTC
I would probably use split or a regex to grab the IP address string (and the last octet range) from each line of the file. Then I would step from the beginning of the range to the end of the range, constructing each IP address and writing it out as I went. So, you'll want to do the following things: Open the file (and do something sensible if that fails) process each line of the file in some sort of loop (maybe a while) extract the IP address substring from each line determine if the IP address is a singleton or a range if it is a singleton, write it out to your file or whatever if it is a range, step through from the start to the end part of the last octet, constructing a series of singleton IP addresses and writing them out. Show us what you have tried, and we'll be glad to help!	[reply]
Re: Need a better way to break out a range of addresses... by sailortailorson (Scribe) on Mar 28, 2007 at 19:52 UTC
This is a good job for an iterator. Mark Jason Dominius talks about this in his book Higher Order Perl, which is supposed to be eventually be available on line, but does not seem to be ready yet. You can look in his code examples, and full text search the book, but you can't just peruse it. Basically you need to implement a closure, which preserves the last value it was called with, and build in a test to end at your desired end value. You may also need to implement a small pattern to let you know where to iterate. Sorry I can't explain more, as I am pressed for time, but here is a working example: put the following in a file called 'Iterator_Utils.pm' `package Iterator_Utils; use base Exporter; use strict; @EXPORT_OK =qw(NEXTVAL Iterator append imap igrep iterate_function filehandle_iterator list_iterator); %EXPORT_TAGS = ('all' => \@EXPORT_OK); sub Iterator (&) { return $_[0] } # 'syntactic sugar' to allow using ' +Iterator { ... }' vice # 'Iterator { sub( ...) }' see Domin +ius, Higher Order Perl, p. 123 sub NEXTVAL { $_[0]->()` [download] keep it in the same dir whence you run the following example code: #!/perl/bin/perl use strict; use Iterator_Utils('NEXTVAL'); $\| = 1; my $bLogToStdout = 1; my $iVerbose = 5; $bLogToStdout = 1; # This example shows how one can generate a range of # URL's with fuskurled numeric arguments. my @aURLsToFuskurl = @ARGV; if ( not @aURLsToFuskurl) { @aURLsToFuskurl = ( "\nEach page of stores index: " , " http://www.hmmm.com/hmmmhmm/cc/main/a_z_index/act/<-{(a..z, +'-','~')}->%2C0%2C0/ccsyn/260" , "\nSome URL forced to display GSL's corresponding to sourceid 1- +20: " , " http://www.hmmm.com/op/~Mens_Mock_Neck_Chevron_Sweater" . ",_Big_Sizes-prod-39296606?slkd=<-{1..20}->" , "\nSame URL, different cluster: " , " http://<-{ qw{www staging marketing} }->.hmmm.com/cc.hmm?ma +in=aprod&act=k24," . "g1,~tea+for+one+teapot+and+cup+set,nover,i1" ); } my @aURLsFuskurled; for (@aURLsToFuskurl) { my $rcFuskurler = fuskurl ( $_ ); while (my $sNextFuskurledURL = NEXTVAL($rcFuskurler)) # # 'kick' the iterator. { push @aURLsFuskurled, $sNextFuskurledURL } } print join "\n", @aURLsFuskurled; sub fuskurl { my $pat = shift; my @tokens = split /(?:\<\-\{\|\}\-\>)/, $pat; for (my $i = 1; $i < @tokens; $i += 2) { #$tokens[$i] = [0, split(/,/, $tokens[$i])]; #print "\n" . $tokens[$i] . "\n"; my @restOfArray = eval $tokens[$i] ; #print join "\n", @restOfArray; $tokens[$i] = [0, @restOfArray ]; } my $FINISHED = 0; return sub { return if $FINISHED; my $finished_incrementing = 0; my $result = ''; for my $token ( reverse @tokens) # reverse to work from right + to left (typically best) { if (ref $token eq '') { $result = $token . $result; #suits reversing @toke +ns } else { my ($n, @c) = @{$token}; $result = $c[$n] . $result; #suits reversing @tokens unless ($finished_incrementing) { if ($n == $#c) {$token->[0] = 0 } else {$token->[0]++; $finished_incrementing = 1 } } } } $FINISHED = 1 unless $finished_incrementing; return $result; } [download]	[reply] [d/l] [select]
Re^2: Need a better way to break out a range of addresses... by sailortailorson (Scribe) on Mar 28, 2007 at 20:17 UTC
So, if you put the example code in a file called Iteratedriver.pl, then invoke it with a couple of your examples (my version of Windows requires me to invoke with 'perl' leading any command with args: `C:\Documents and Settings\martyg\My Documents\my scripts\Perl>perl Ite +ratedriver.pl "172.<-{19..21}->.254.<-{2..3}->" "192.168.1.<-{1..3}-> +" 172.19.254.2 172.19.254.3 172.20.254.2 172.20.254.3 172.21.254.2 172.21.254.3 192.168.1.1 192.168.1.2 192.168.1.3 C:\Documents and Settings\martyg\My Documents\my scripts\Perl>` [download]	[reply] [d/l]
Re: Need a better way to break out a range of addresses... by pgor (Beadle) on Mar 28, 2007 at 21:04 UTC
A recursive version (loading your IP specs into `@ips` left as a trivial exercise): `for my $ip ( @ips ) { print "$_\n" for expandQuads( split( /\./, $ip ) ); } sub expandQuads { my @quads = @_; my $first = shift(@quads); my @newQuads = ( $first =~ /(\d+)-(\d+)/ ) ? ($1..$2) : ($first); return ( @quads ) ? map { my $str = $_; map { "$str.$_" } expandQuads(@quads) } @newQuads : @newQuads; }` [download] pg	[reply] [d/l] [select]
Re: Need a better way to break out a range of addresses... by Cristoforo (Curate) on Mar 29, 2007 at 02:27 UTC
An example using Set::CrossProduct `#!/usr/bin/perl use strict; use warnings; use Set::CrossProduct; while (<DATA>) { chomp; s/\s*#.+//; my @data; while (/([^.]+)/g) { if ($1 =~ /(\d+)-(\d+)/) { push @data, [$1..$2]; } else { push @data, [$1]; } } for (Set::CrossProduct->new( \@data )->combinations) { print join(".", @$_), "\n"; } } __DATA__ 172.17.119.2 # Comments are here... 172.17.119.4-5 # Comments are here... 172.19-21.254.2-3 # Comments are here... 192.168.1.1-3 # Comments are here...` [download] Chris	[reply] [d/l]
Re^2: Need a better way to break out a range of addresses... by brian_d_foy (Abbot) on Mar 29, 2007 at 04:18 UTC
Nice use of Set::CrossProduct! Note, however, that if the data set is large you might want to stick with the `get` method. The `combinations` method generates all of the combinations at once and returns a big list, whereas `get` just gives a new combination without creating the big list internally. There's no need for fancy HOP magic or nested loops (that you have to think about, anyway :) `#!/usr/bin/perl use strict; use warnings; use Set::CrossProduct; while( <DATA> ) { chomp; s/\s#.//; my @b = map { [ m/(\d+)-(\d+)/ ? $1 .. $2 : $_ ] } split /\./; my $cross = Set::CrossProduct->new( \@b ); while( my $array_ref = $cross->get ) { print join( ".", @$array_ref ), "\n"; } } __END__ 172.17.119.2 # Comments are here... 172.17.119.4-5 # Comments are here... 172.19-21.254.2-3 # Comments are here... 192.168.1.1-3 # Comments are here...` [download] And, just for giggles and since I already wrote it before I saw your post, my version of the same program that you wrote, but mostly as a list pipeline: `#!/usr/bin/perl use strict; use warnings; use Set::CrossProduct; $, = "\n"; print map { chomp; s/\s(#.)?$//; map { join ".", @$_ } Set::CrossProduct->new( [ map { [ m/(\d+)-(\d+)/ ? $1 .. $2 : $_ ] } split /\./ ] )->combinations; } <DATA>; __END__ 172.17.119.2 # Comments are here... 172.17.119.4-5 # Comments are here... 172.19-21.254.2-3 # Comments are here... 192.168.1.1-3 # Comments are here...` [download] -- brian d foy <brian@stonehenge.com> Subscribe to The Perl Review	[reply] [d/l] [select]
Re^3: Need a better way to break out a range of addresses... by Cristoforo (Curate) on Apr 02, 2007 at 15:36 UTC
It wasn't so easy to see the solution at first. I shut the computer off and went for an easy chair to try and figure it out. I could see that I wanted all combinations of the values that expressed a range but I didn't know how to generate them. Then, it occurred to me that the values that were just one number were also a set - a set of one. So then, the solution came to me! ~~CPAN - what a treasure!~~	[reply]
Re: Need a better way to break out a range of addresses... by gloryhack (Deacon) on Mar 28, 2007 at 19:52 UTC
The easy way: use Net::IP.	[reply]
Re^2: Need a better way to break out a range of addresses... by Fletch (Bishop) on Mar 28, 2007 at 19:57 UTC
While that module does work with ranges of addresses (`'20.34.101.207 - 201.3.9.99'` is the example in its documentation), doesn't look to deal with the syntax the OP is dealing with (with embedded ranges for the separate octets). `DB<1> x Net::IP->new( "10.0.2.3-5" ) empty array + DB<2> x Net::IP->new( "10.0.2.3-10.0.2.5" ) 0 Net::IP=HASH(0x198db00) 'binip' => 00001010000000000000001000000011 'ip' => '10.0.2.3' 'ipversion' => 4 'is_prefix' => 0 'last_bin' => 00001010000000000000001000000101 'last_ip' => '10.0.2.5'` [download]	[reply] [d/l]
Re^3: Need a better way to break out a range of addresses... by gloryhack (Deacon) on Mar 28, 2007 at 20:42 UTC
True, and perhaps I should have elaborated as did grinder in the tentative solution presented below.	[reply]
Re: Need a better way to break out a range of addresses... by johngg (Canon) on Mar 31, 2007 at 23:18 UTC
This approach uses `splice`ing and `map`s to expand ranges in any of the four parts of an IP address, building an AoA for each data item and passing it along to the `print`. use strict; use warnings; print map { qq{@{ [ join q{.}, @$_ ] }\n} } map { my @quads = ( $_ ); foreach my $part ( 0 .. 3 ) { foreach my $offset ( reverse 0 .. $#quads ) { next unless $quads[$offset]->[$part] =~ m{(\d+)-(\d+)}; splice @quads, $offset, 1, map { my @quad = @{ $quads[$offset] }; splice @quad, $part, 1, qq{$_}; [ @quad ]; } $1 .. $2; } } @quads; } map { [ ( split m{\.\|\s} )[0 .. 3] ] } <DATA>; __END__ 172.17.119.2 # Comments are here... 172.17.119.4-5 # Comments are here... 172.19-21.254.2-3 # Comments are here... 192.168.1.1-3 # Comments are here... [download] I'm a bit late to the party but it is an interesting problem and the posted solutions have introduced modules I'd not encountered before. Cheers, JohnGG	[reply] [d/l] [select]
Re: Need a better way to break out a range of addresses... by NetWallah (Canon) on Mar 29, 2007 at 19:42 UTC
Since you need to generate valid IP addresses - rejecting nonsense IP's like 192.168.3.256, and following up on grinder's quest, I offer the following code, that uses Net::IP: use strict; use Net::IP; while (<DATA>){ chomp; generate_range($_); } sub generate_range{ my $range =shift; $range =~s/#.//; print "\n------ $range -------\n"; my ($lo,$hi); for (split /\./,$range,4){ my ($l,$h) = split /-/; ## print qq[ $_; L=$l H=$h\n] ; $lo .= "$l."; $hi .= ($h \|\| $l ) . "."; } chop for ($lo, $hi); # Drop tralining dots print " $lo - $hi; \n"; my $ip = new Net::IP ("$lo - $hi") \|\| die "No IP$!"; print ( "Number of IPs=" . $ip->size() . "\n"); my $count=0; do { print $ip->ip(), "\t"; } while (++$ip && $count++ <= 20); print "\n"; } __DATA__ 172.17.119.2 # Comments are here... 172.17.119.4-5 # Comments are here... 172.19-21.254.2-3 # Comments are here... 192.168.1.1-3 # Comments are here... [download] Limiting the output to 20 printed IP's, the output looks like : ------ 172.17.119.2 ------- 172.17.119.2 - 172.17.119.2 ; Number of IPs=1 172.17.119.2 ------ 172.17.119.4-5 ------- 172.17.119.4 - 172.17.119.5 ; Number of IPs=2 172.17.119.4 172.17.119.5 ------ 172.19-21.254.2-3 ------- 172.19.254.2 - 172.21.254.3 ; Number of IPs=131074 172.19.254.2 172.19.254.3 172.19.254.4 172.19.254.5 172.19 +.254.6 172.19.254.7 172. 19.254.8 172.19.254.9 172.19.254.10 172.19.254.11 172.19 +.254.12 172.19.254.13 172. 19.254.14 172.19.254.15 172.19.254.16 172.19.254.17 172.19 +.254.18 172.19.254.19 172. 19.254.20 172.19.254.21 172.19.254.22 172.19.254.23 ------ 192.168.1.1-3 ------- 192.168.1.1 - 192.168.1.3 ; Number of IPs=3 192.168.1.1 192.168.1.2 192.168.1.3 [download] I verified the correct generation of the IP sequence: 172.19.255.0 .. 172.19.255.1 "Choose a job you like and you will never have to work a day of your life" - Confucius*	[reply] [d/l] [select]
Re: Need a better way to break out a range of addresses... by bsdz (Friar) on Mar 29, 2007 at 22:46 UTC
Wow! I think I missed this one :( Still, here is yet another alternative using Perl out of the box. No doubt, a master monk can probably shorten it even more. `use strict; use warnings; my @a = map { s/\s(#.)?$//g; $_ } <DATA>; my @b; while (local $_ = shift @a) { if (m/\D(\d+)-(\d+)(?:\D\|$)/) { my $range = "$1-$2"; foreach my $i ($1..$2) { local $_ = $_; s/(\D)$range(\D\|$)/$1$i$2/; push @a, $_; } } else { push @b, $_; } } print join("\n", @b); __END__ 172.17.119.2 # Comments are here... 172.17.119.4-5 # Comments are here... 172.19-21.254.2-3 # Comments are here... 192.168.1.1-3 # Comments are here...` [download] Update: Slightly saner while loop using localized stack. Update 2: Removed unnecessary localization of $1, $2. Thanks to ikegami++	[reply] [d/l]
Re^2: Need a better way to break out a range of addresses... by RMaxwell (Novice) on Apr 02, 2007 at 19:45 UTC
Hello, I used your solution and it worked like a charm!! Thanks for everyones help on this, there were a lot of cool solutions derived for my problem! Thanks, Robert Maxwell	[reply]