I'd get rid of the nested sub altogether and take advantage of Perl's array handling to clean up the loop handling. Using early exits as soon as an error is detected cleans up the code and reduces execution time (although that's unlikely to be an issue). Consider:

use strict;
use warnings;

open TXT, "<masks.txt" or die "Can't open masks.txt\n";

while (<TXT>) {
    chomp;
    my @words = split (/ +/);
    print "Checking mask $words[2]\n";
    my $maskvalid = ValidateMasks ($words[2]);
    print "Mask is $maskvalid\n\n";
}

sub ValidateMasks {
    my $mask = shift;
    
    return 0 unless defined $mask;
    my @octets = split /\./, $mask;

    print "\toctets array is @octets\n";

    return 0 unless 4 == @octets; # Bad if too few parts
    
    my $flagvalue = 0;

    for my $octet (@octets) {
        return 0 if $octet > 255;     # bad if value is above 255
        print "\tchecking octet $octet\n";
        next if ($octet eq $flagvalue);    # good if equal to flag val
+ue
        print "\t\tUnequal: oc is $octet, flagvalue is $flagvalue\n";
        
        # bad if this is a second non-zero and it's not equal to 255.
        return 0 if $flagvalue; 
        $flagvalue = 255;    # All further octets after this one must 
+be 255

        # This is the first non-zero octet.
        # Should be equal to a power of 2 minus 1
        # X is a power of 2 if (X & X-1) = 0
        # $octet + 0 forces value to number vice string
        return 0 unless ($octet + 0) & ($octet + 1);
    }
    
    return 1;
}
[download]

Oh, and don't use prototyped subs, they generally don't do what you expect (see Gratuitous use of Perl Prototypes).