My assumptions are:

a) $urls will inherit only the unique elements of the @hrefs array

That's your mistake. When you iterate over an array using for, the indexing variable ($urls) will take on each value in turn regardless of whether it is unique or not.

The simplest way to ensure uniqueness (assuming that ordering is not important to your application) is to use a hash.

my @hrefs = ...;
my %uniqHrefs; 

### This will create one entry in %uniqHrefs for each unique value in 
+@hrefs;
@uniqHrefs{ @hrefs } = ();

my $res = Net::DNS::Resolver->new;  
 
foreach $urls ( keys %uniqHrefs ) {  
 
    my $query = $res->query($urls, "A");  
...
[download]

If you need to retain ordering, then you'll need something slightly different but that doesn't seem to be the case.

To follow up to BrowserUk's suggestion, you could also use a '%seen' hash (there's nothing magical about the name - it's just an indicator of its function.) This also processes the URLs in order.

my %seen;
for $urls (@hrefs){  
    next if $seen{$urls}++;
    my $query = $res->query($urls, "A"); 
    ...
[download]

Each new URL becomes a key in '%seen'; if a key identical to the current one already exists, we return to the top of the loop.

-- 
Human history becomes more and more a race between education and catastrophe. -- HG Wells

Hi mischief,

I don't think this is correct - I am trying to do things this way for that reason specifically.

If I didn't make the URLs unique before passing them to the Net::DNS::Resolver module I would be doing even more duplication (some duplication is unavoidable as you correctly pointed out).

By resolving only unique URLs I am minimizing the amount of DNS resolution that is required. It would be wasteful to resolve the exact same URL multiple times as it would always yield the same result.

Thanks for the reply mischief,

I actually didn't make that assumption. I could have easily not worried about things being unique until I got to the IP values - but by doing that I am wasting processing power and bandwidth by doing a DNS lookup on the same URL multiple times.

This is why I decided to make sure I pass a unique set of hostnames to the Net:DNS::Resolver and then run the results (IPs (including duplicates)) through the Array::Unique module again.

In fact if you try the script as is and compare the regex results before and after passing through the Array:Unique module you will see I am saving a large amount of duplication.

That's better than what I thought I read, though you could cut the resolver overhead just as much by checking against each hostname before you look it up to see if it's been looked up before. This sort of task is just begging for a hash. That would use one loop instead of two, and probably be simpler to follow.

my %looked_up;
my @urls = qw( list of URLS however you got them );
my @ips;

foreach ( @urls ) {
    my $hostname = extract_hostname_from_url( $_ );
    unless ( exists $looked_up{ $hostname ) {
        my $packed_ip = gethostbyname( $hostname );
        if (defined $packed_ip) {
            $ip_address = inet_ntoa($packed_ip);
            push @ips, $ip_address;
            $looked_up{ $hostname } = 1;
        }
    }
}

# do with @ips whatever you were going to do with them
[download]

This retries hostname lookups that fail. You could change that easily by moving the autovivifying hash element assignment outside the if block for the packed IP address being defined.

the sub extract_hostname_from_url is left as an exercise.

One thing that you should always be very-aware of is CPAN: the rather-vast library of Perl software. Without exception, the first thing that you should do, when contemplating a “new” task, is to check CPAN to see if it's already been done. It probably has.