in reply to Re: Crawling Relative Links from Webpages
in thread Crawling Relative Links from Webpages

Thanks for your reply.

Well OK. The point is how do you determine $url? In this case, the $url is "http://dspace.mit.edu" and it is not at all obvious from the webpage (looking at the source) how one would say that this is the server. I have a million different kinds of such webpages from different servers. I need a method that is generic enough to work with all of them.

Any suggestions anyone?

Andy

  • Comment on Re^2: Crawling Relative Links from Webpages

Replies are listed 'Best First'.
Re^3: Crawling Relative Links from Webpages
by BrowserUk (Patriarch) on May 08, 2010 at 01:42 UTC

    Something like: 

    my $uri = $mech->uri;
my( $server ) = $url =~ m[(^http://[^/]+)/];
...
my $pdfurl = $server . $link;

    [download]

    Note: There probably is some way of getting the appropriate portion of the url from URI without resorting to regex, but I've never worked out how.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
      uri returns a URI object, so $mech->uri->host or $mech->uri->ihost
[reply]
[d/l]
[select]

        I know. But how do you get the bit the OP needs? Not like this: 

        perl -MURI 
-E"$u=new URI('http://dspace.mit.edu/handle/1721.1/53720'); say $u->ho
+st"
dspace.mit.edu

        [download]

        Nor any of these: 

        c:\test>perl -MURI -E"my $u=new URI('http://dspace.mit.edu/handle/1721
+.1/53720'); say $u->authority"
dspace.mit.edu

c:\test>perl -MURI -E"my $u=new URI('http://dspace.mit.edu/handle/1721
+.1/53720'); say $u->path"
/handle/1721.1/53720

c:\test>perl -MURI -E"my $u=new URI('http://dspace.mit.edu/handle/1721
+.1/53720'); say $u->fragment"


c:\test>perl -MURI -E"my $u=new URI('http://dspace.mit.edu/handle/1721
+.1/53720'); say $u->opaque"
//dspace.mit.edu/handle/1721.1/53720

c:\test>perl -MURI -E"my $u=new URI('http://dspace.mit.edu/handle/1721
+.1/53720'); say $u->canonical"
http://dspace.mit.edu/handle/1721.1/53720

        [download]
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom