LWP question

cajun has asked for the wisdom of the Perl Monks concerning the following question:

If you were using LWP to suck down a www page, lets say a member search on yahoo for members with perl mentioned in their profile. The following is what is sent to yahoo to do the search:

http://members.yahoo.com/interests?.oc=t&.kw=perl&.sb=0

Now let's say that yeilds more than one page when run in a browser. How is it done in LWP to instruct the www server not to just show bits at a time, but show the entire listing at once. Hopefully this makes sense.

Comment on LWP question

Replies are listed 'Best First'.
Re: LWP question by eg (Friar) on Jan 26, 2001 at 13:08 UTC
There's no general solution to this sort of problem. You either need to find the parameter (if it exists) that specifies how many results are shown per page or you need to write your user-agent to pull down successive pages (for you yahoo page it's passing the ".ss" parameter.) One hint is to look on the "Advanced Search" page to see what are the possible parameters you can use. In this case, it doesn't look like there's an option to specify the page length.	[reply]
Re: LWP question by clemburg (Curate) on Jan 26, 2001 at 14:18 UTC
While starting to "roll your own", you might want to take a look at WWW::Search. Christian Lemburg Brainbench MVP for Perl http://www.brainbench.com	[reply]
Re: LWP question by wardk (Deacon) on Jan 26, 2001 at 21:27 UTC
Use LWP to Grab an example multipage reply and get a feel for their page syntax, it should be repeatable (at least until they change it), and provide some sort of Reugular Expression opportunity identifying and extracting the "next page" link. Then use LWP to grab that, until you get them all. Should be a very fun exercise. good luck.	[reply]
Re: LWP question by cajun (Chaplain) on Jan 27, 2001 at 13:20 UTC
Thanks to all for the suggestions. I was afraid there wasn't a really easy method. I hadn't thought of the WWW::Search modules. Thanks, I'll check all of these out.	[reply]