I am working on my first perl script (no programming exp. at all) and am in well over my head!

Here is what I am trying to accomplish:

1. Grab a index.html page
2. Parse out all of the URLs
3. Create a unique list of the URLs
4. Convert the unique list of URLs to a unique list of IP addresses.

So far I have 3 1/2 of this done. I am able to print out a unique list of URLs (stored in an array) - I can convert URLs to IP addresses but somehow I am loosing the unique portion of the equation when I print out the resolved urls.

I have never done any programming before - so I suspect it may be syntax related but I can't seem to find the problem. I have read through the docs on the modules I am using but still had some lingering questions. I posted at Perl Guru forums but the repose I got was - ask the monks!

Line 62 is a comment explaining the closest I have come and following on 64-67 is what prints out the full list of IPs (but with duplicates).

In the comments you can see I made notes of where I can prove the array is returning unique elements. (I tested by printing the array).

I may have made a bad assumption however. When I start using the Net::DNS module to resolve the URLs I do the following: 


my $res = Net::DNS::Resolver->new;  
 
foreach $urls (@hrefs) {  
 
my $query = $res->query($urls, "A");

[download]
My assumptions are:

a) $urls will inherit only the unique elements of the @hrefs array
b) that I can only query a scalar variable...not an array using Net::DNS.

Any help would be greatly appreciated!

Cheers!

Here is the complete script: 

#!/usr/bin/perl 
use LWP::Simple; 
use Array::Unique; 
tie @hrefs, 'Array::Unique'; 
tie @ips, 'Array::Unique'; 
use Net::DNS; 
 
 
# Store the output of the web page (html and all) in the $content vari
+able 
# 
 
print "URL to Parse? "; 
$a = <STDIN>; 
chop $a; 
$b = "http://"; 
$c = $b . $a; 
$" = \n; 
my $content = get("$c"); 
if (defined $content) 
{ 
 
#$content will contain the html associated with the url mentioned abov
+e. 
#print $content.""; 
} 
else 
{ 
#If an error occurs then $content will not be defined. 
print "Error: Get failed"; 
} 
 
# 
# Parse ALL URLs from the .html page and store all unique entries in t
+he @hrefs array 
# 
 
$content =~ s/\'//g; 
$content =~ s/\+//g; 
@hrefs = ($content =~ m/:\/\/\"*([\w\-]+\.[\w\-\.]+?)[\"|\/|\s+]/ig); 
 
 
# 
# Resolve the URLs stored in the array 
# 
 
 
my $res = Net::DNS::Resolver->new; 
foreach $urls (@hrefs) { 
# 
## things are unique here!! 
# 
my $query = $res->query($urls, "A"); 
# 
## things are unique here!! 
# 
  if ($query) { 
      foreach $rr (grep { $_->type eq 'A' } $query->answer) { 
# 
##  At this location uniqueness is not preserved 
# 
#     print ($rr->address . ' ');  # This prints out many duplicate IP
+ addresses in a scalar variable 
 
# 
#  This was one of my attempts to get things working 
#  This is the closest to what I want to do but the list is not unique
+. 
# 
@ips = ($rr->address); 
foreach $uips (@ips)  { 
print $uips . "\n"; 
} 
 
      } 
 
  } 
  else { 
      warn "query failed: ", $res->errorstring, "\n"; 
  } 
} 
# 
# for troubleshooting - this print statement is unique 
# 
#print $urls . "\n"; 
 
# 
## More testing below - didn't work 
# 
 
# 
# Add unique IPs to array @ips 
# 
 
#foreach $unique (@ips) { 
 
# print "@ips"; 
# print $unique . "\n"; 
 
#}

[download]

In reply to issues maintaining uniqueness by jkstraw

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.