Mech provides for this. There should be no need for HTML::LinkExtor or any hackery. :)

use warnings;
no warnings "uninitialized";
use strict;
use WWW::Mechanize;

use MIME::Types;
my $mt = MIME::Types->new;

my $mech = WWW::Mechanize->new();
$mech->get("http://dspace.mit.edu/handle/1721.1/53720");

for my $link ( $mech->links() )
{
    my $uri = $link->url_abs();
    print $uri, $/
        if $mt->mimeTypeOf($uri->path) eq "application/pdf";
}

# http://dspace.mit.edu/bitstream/handle/1721.1/53720/MIT-CSAIL-TR-201
+0-018.pdf?sequence=1
[download]

(Update: please only use these tools under the Terms of Service of any given site. The better hackers behave, the more likely we are to get open resources facing us.)

(Update #2: I should add the .pdf is a fairly artificial way to check. It could serve something else and something like /taco could serve a PDF. You'll have to do actual GET|HEAD requests to find out the reputed mime type of any given resource and verify it after receiving it.)

OK so maybe I am missing something here, because I am just unable to understand what's being said :(

$mech above uses a hard coded link, which would of course work for this page. What about those from other domains (say "xyz.com")?

How do I make the method generalizable?

There is only one hard-coded address in the code:

my $mech = WWW::Mechanize->new();
$mech->get("http://dspace.mit.edu/handle/1721.1/53720");
[download]

If you want to make that variable, maybe you want to pass the starting link from the command line? It will then be available via @ARGV:

my $mech = WWW::Mechanize->new();
warn "Fetching $ARGV[0]\n";
$mech->get($ARGV[0]);
[download]

Call it as

perl -w listanand.pl http://google.com
[download]

The pdf link on the example page is not relative to the current page. It starts with /, so is absolute path relative to the current server, so combining with base() isn't going to work.

You need to combine it with the server to form the correct url, but Mech doesn't break that out for you. It will give you a URI, but that's documented in alien and so I have never been sure if it can give you the root address of the server or not. I've always used:

my( $server ) = $url =~ m[(^http://[^/]+/)];
[download]

Thanks for your reply.

Well OK. The point is how do you determine $url? In this case, the $url is "http://dspace.mit.edu" and it is not at all obvious from the webpage (looking at the source) how one would say that this is the server. I have a million different kinds of such webpages from different servers. I need a method that is generic enough to work with all of them.

Any suggestions anyone?

Andy

Something like:

my $uri = $mech->uri;
my( $server ) = $url =~ m[(^http://[^/]+)/];
...
my $pdfurl = $server . $link;
[download]

Note: There probably is some way of getting the appropriate portion of the url from URI without resorting to regex, but I've never worked out how.

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

RIP an inspiration; A true Folk's Guy