dsheroh has asked for the wisdom of the Perl Monks concerning the following question:

I'm working on a project which involves pulling some data from Google search results. Since I know Google doesn't like people scraping their search result pages, it was off to CPAN to see what might be available there. DBD::Google looked particularly promising and it (like most other Googly modules) is based on Net::Google, which in turn uses the Google SOAP API. So, next stop, get a Google SOAP API key.

And then I came to a screeching halt. "As of December 5, 2006, we are no longer issuing new API keys for the SOAP Search API... Depending on your application, the AJAX Search API may be a better choice for you instead. It tends to be better suited for search-based web applications..."

Well, I'm not doing a search-based web application, I'm looking to grab some document titles/URLs and stuff them into an email message. A quick review of the AJAX Search API examples looks an awful lot like it doesn't do anything that's particularly useful to me - I probably could work up a way to use it, but I'd just be scraping those results instead of the results from the normal search form.

Is there a new Google API which replaces the functionality of the apparently-discontinued SOAP API or has Google basically decided to say "screw you" to anyone who wants to use search results in ways other than displaying them on a web page? If there is a new API, do any Perl modules exist to provide an interface to it (yet)?

In the end, I can scrape the standard page if that's what's needed, but I would, of course, prefer to avoid incurring the wrath of Google...

Replies are listed 'Best First'.
Re: Net::Google replacement?
by pemungkah (Priest) on Sep 12, 2007 at 02:00 UTC
    Well, there's always Yahoo: http://developer.yahoo.com/search/.

    Disclaimer: I work there. Then again, that means you know who to complain to if you don't like the Perl interface, or if you have patches or suggestions...