in reply to Loading A Site Into An Array

Coincidence? No such thing; I've been working on a similar problem (building from code a freind provided)

<xmp> #!/usr/local/bin/perl use LWP::Simple; open (IN, "sorturls"); @urls = <IN>; foreach $url (@urls){ chop($url); $doc = get "$url"; push (@downloaded_pages, $doc); }; close(IN); </xmp>

Your question is a little unclear, though. Do you want the whole page to into a single element of an array> 'Cause, that's what the above will do for each page named in "sorturls" (which has a single url, including "http://" on each line)

-T.McWee

mcwee@m-net.arbornet.org

Replies are listed 'Best First'.
RE: Re: Loading A Site Into An Array
by Anonymous Monk on Mar 10, 2000 at 02:38 UTC
    To explain my situation better, I'll give some more details. I would like to make a script that functions similar to anonymizer.com and loads a whole website into an @array. I then plan to replace all <a href="www.yahoo.com"> tags with <a href="http://www..com/route.cgi?www.yahoo.com"> So that they go through the script first. The website that it loads could be any website on the internet. Any help would be appreciated, and the help already given is greatly appreciated as well.
[reply]
[d/l]
[select]
      You might want to take a look at Randal Schwartz's anonymous proxy server (created for one of his WebTechniques columns).

      It doesn't try to do what you're doing (replacing links to go through a CGI script), but rather uses your browser's built-in ability to use a proxy server.

      Anyway, though, for what you asked about, take a look at HTML::LinkExtor (a subclass of HTML::Parser). 

      perldoc HTML::LinkExtor
perldoc LWP::UserAgent

      [download]
      You can use LWP to fetch the web page, then extract the links, then replace each link by a modified version of itself that routes the user through your program.
[reply]
[d/l]

In Section Seekers of Perl Wisdom