Re^2: Cannot retrieve HTML for some pages with LWP

Thanks for your reply. I have the same problem with LWP::UserAgent. I understand the problem. However, how to retrieve web data as a browser, i mean, to realize to action of avoiding automatic access of the web page and fix them.

Comment on Re^2: Cannot retrieve HTML for some pages with LWP

Replies are listed 'Best First'.
Re^3: Cannot retrieve HTML for some pages with LWP by marnanel (Beadle) on May 27, 2005 at 17:28 UTC
Firstly, please be aware of the issues surrounding accessing Google's site in contravention of their terms of service. It might be easier for you to use Google's own web APIs, assuming they work for Google Scholar. Look into Net::Google for examples which use ordinary Google search. If after all you want to scrape Google Scholar, you may have some luck modifying WWW::Scraper::Google.	[reply]