Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask

fastest way to get a web page

by chorg (Monk)
on Oct 26, 2002 at 22:41 UTC ( #208287=perlquestion: print w/replies, xml ) Need Help??

chorg has asked for the wisdom of the Perl Monks concerning the following question:

I am noticing speed issues with LWP::UserAgent... is there a high speed way to get a webpage with perl?
forgot -- i want to do this "in process" and not shell out if i can help it .. i am trying a thread experiment with 5.8
"Intelligence is a tool used achieve goals, however goals are not always chosen wisely..."

Replies are listed 'Best First'.
Re: fastest way to get a web page
by Revelation (Deacon) on Oct 27, 2002 at 01:58 UTC
    Bio::Root::HTTPget is an alternape LWP::UserAgent mechanism, but I'm not sure about the benchmarks. It looks pretty small, so it should be pretty quick.

    Kanji already mentioned HTTP::GHTTP

    However, my advice is to check your bottlekneck first. Are you sure you own bandwidth isn't limmiting you? If you have a 56k connection, and are downloading a 200kb file, then that's probably the real problem. On the other hand, it's possible it is the module, and I guess I'd use one of these two (I'd adapt Bio::Root::HTTPget, personally.)
    Gyan Kapur
Re: fastest way to get a web page
by BrowserUk (Patriarch) on Oct 26, 2002 at 22:53 UTC

    I think you'll need to identify a little more about what the "speed issues" are. Some timings, comparisons, maybe show some code before we would be able to make any useful suggestions.

    Nah! Your thinking of Simon Templar, originally played by Roger Moore and later by Ian Ogilvy
      Well I just am looking for XS modules that do what LWP does essentially. I checked out Curl::easy -- which uses libcurl to grab pages ... was wondering if there are others?

      Or to put it a different way, what alternatives are there to using LWP to get/post webdocuments inprocess?
      "Intelligence is a tool used achieve goals, however goals are not always chosen wisely..."

        Offhand, I can only think of HTTP::GHTTP (alt.), which even provides some comparative benchmarks ...

        Benchmark: timing 1000 iterations of ghttp, lite, lwp... ghttp: 8 wallclock secs ( 0.96 usr + 1.16 sys = 2.12 CPU) lite: 21 wallclock secs ( 3.00 usr + 3.44 sys = 6.44 CPU) lwp: 18 wallclock secs ( 9.76 usr + 1.59 sys = 11.35 CPU)

        It may be a little too "lite" for your needs, though, in which case ... perhaps rolling your own with IO::Socket or -gasp- the lower level Socket and friends?


Re: fastest way to get a web page
by true (Pilgrim) on Oct 27, 2002 at 20:17 UTC
    An alternative would be to use LWP under mod_perl. If you are only testing one http request, you may not notice a difference. But if you are doing several in a loop, Mod_perl is worth the time to install and run. I timed a HTTP request for 500 static html pages on my RHL server with a perl script. Without mod_perl, it took me 39.25 seconds to request 500 pages. With mod_perl, it took 3.34 seconds.
    I saved 36 seconds with mod_perl.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://208287]
Approved by George_Sherston
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (4)
As of 2023-10-04 06:56 GMT
Find Nodes?
    Voting Booth?

    No recent polls found