Sibling monks, give ear unto my tale I pray.

I'm working on a script that will crawl a number of newspaper sites and pull out articles that match some search criteria. So at the moment I have a file, "papers.txt" of URLs chosen because they are teh main link pages for the articles I'm interested in, and I'm going through them like this:

use LWP::Simple;
open PAPERS, "papers.txt";
while (<PAPERS>) {
    $page = $_;
    $html = get $page or next;
    while ($html =~ s/<a href="(.*?)".*?>(.*?)<//is) {
        my $url = $1;
        my $headline = $2;
        if ($headline =~ /$regexp/i) {   # defined elsewhere
            my $article;

# get the article - but HOW?

            SaveArticle($article,$headline,$url);
        }
    }
}
[download]

Now, this works as far as it goes. But obviously it doesn't go all the way. What I want is to take $url and feed it again to LWP::Simple to pull out the text of the article which I can then parse and save. And I can't rely on using $url because it's almost certainly a relative URL.

My first try was:

while (<PAPERS>) {
    $page = $_;
    $html = get $page or next;
    my $path = $_;
    $path =~ s#^(.*)/.*?$#$1#;
    while ($html =~ s/<a href="(.*?)".*?>(.*?)<//is) {
        my $url = $1;
        my $headline = $2;
        if ($headline =~ /$regexp/i) {
            $url = $path . $url;
            my $article = get $url || 'goom';
            SaveArticle($article,$headline,$url);
        }
    }
}
[download]

But quite often this fails for one reason or another... perhaps because the page I'm on is on a different path from the page I want. What I'd really like is to get Perl to do exactly what I'd do, that is, click on the link, and let the page itself take care of the question whether it's a relative link and what it's relative *to*.

Is there a neat way to do this?

I had a look at URI::URL::full_path, but as far as I can see this wd save me some time *if* I already knew what I needed to create the absolute URL, but I don't.

It may be that I am betraying some ignorance of how URLs work, and if so I apologise for asking a dim question an solicity enlightenment.

§ George Sherston