Repeat as neccessary.

my $time = localtime;
1 while ($time + 30 > localtime); 
`shutdown -r now`;
[download]

Of course, you may not be able to do this unless you run it as root, but that should give you some idea.

cLive ;-)

Go find an http proxy in Perl. (merlyn's columns would be a good place to start.) Modify it to parse the documents passing through to find links in them. (HTML::LinkExtor may help here.) Then set a copy of IE to use that as your proxy server. Use the OLE modules to drive IE around the web.

Voila! :-)

If you just need an answer to a question, this might be acceptable. But I (obviously) wouldn't put this into production. (Says the man who once had a temporary hack which used IE to produce PDFs left in production for half a year...)

This is a pretty simple-minded example how you might do this using LWP::UserAgent and HTML::Parser:

#!/usr/bin/perl -w

use strict;

use LWP::UserAgent;
use HTML::Parser;
use URI;

my $starturl = shift || die "No url supplied\n";

my $baseuri = URI->new($starturl);

my @urls ;

push @urls,$starturl;

my $agent = new LWP::UserAgent;
my $parser = HTML::Parser->new(api_version => 3,
                               start_h => [\&start ,"tagname, attr"]);

$agent->agent("Gelzilla/666"); 

while( my $url = shift @urls)
{
  my $request = new HTTP::Request 'GET' => $url;

  my $result = $agent->request($request);

  if ($result->is_success) 
  {
      print $result->as_string;
      $parser->parse($result->content);
  } 
  else 
  {
     print "Error: " . $result->status_line . "\n";
  }
}

sub start
{
   my($tag,$attr) = @_;
   if ($tag eq 'frame' ) 
   {
      my $thisuri = URI->new($attr->{src});
      push @urls, $thisuri->abs($baseuri);
   }
}
[download]

Of course you are going to have to make your own arrangements for displaying things properly :)

/J\

Wow, that was way too easy :)

Major ++ for that. I was expecting a partial answer but you pushed it nearly all the way to the end. This is very close to what I wanted. I have to do more tests to see if it will handle all possible cases (or at least as many as I can come with), but it looks like a great solution.

Thanks.

---

I admit it, I am Paco.

these pages that the crawler will go to may have 
the link on any of a variety of ways. It may be in a
 frame, it may be generated by the javascript, or
 it may have meta-refresh
[download]

It seems from what you wrote above, that what you're actually saying is that the page may contain the URL in any form, not a link as such.

A Meta-Refresh would take you to the other URL, but isn't technically a link. Also, a JavaScript statement like window.location='http://URL.com' would take a JavaScript-enabled browser to the new URL, but again, isn't a link.

Both these examples require that the URL, as a string, be present, but a link requires that it be present and surrounded with the right HTML code.

Apart from recursively testing frames, this shouldn't be hard to do.

--
Weaselling out of things is important. It's what separates us from the animals ... except the weasel.

Maybe the Inline::Java family of modules can help with rendering the javascripted bits.

Warning, I've never tried it.

Update: Inline::Java should be able to to parse javascript syntax. You would need to provide it a navigator class, OnThisNThat methods, etc, which I think could be written in Perl with that module. Inline is a remarkable namespace.

After Compline,
Zaxo

Java and JavaScript are completly different languages. However, Netscape Navigator (I don't know about mozilla) has fairly good bridging.

Also, it may be possible to perlify Mozilla's JavaScript engine. (Which is designed to be used as a library seperate from mozilla, IIRC.) That would certianly be a worthwhile, and probably rather difficult, project.

Update: I actualy did Recall Correctly. The javascript interpreter mozilla uses is called JSRef, AKA SpiderMonkey. There is also Rhino, which is pure Java, and thus probably harder for you to use.

Update: Fixed links.

TACCTGTTTGAGTGTAACAATCATTCGCTCGGTGTATCCATCTTTG ACACAATGAATCTTTGACTCGAACAATCGTTCGGTCGCTCCGACGC

Even better: There's already support for using JavaScript in Perl and vice-versa; JS.pm and PerlConnect are in the mozilla CVS.

However, they don't work (for me, at least). It could well be trivial; it looks like some symbols aren't getting exported. I'm looking into it right now.

TACCTGTTTGAGTGTAACAATCATTCGCTCGGTGTATCCATCTTTG ACACAATGAATCTTTGACTCGAACAATCGTTCGGTCGCTCCGACGC