Re^2: extracting data from HTML

Is it only me that has this?

I tried to get it all working, from the example with my $xml = HTML::HTML5::Parser->load_html... but ofcourse my test website had to come back with an error. my $xml = HTML::HTML5::Parser->load_html... doesn't handle options I figured and had to use $parser->parse_html_file($URL,{'ignore_http_response_code => 1}). However, ofcourse this happens to me... the user_agent was not accepted and returned a HTTP-406 error

after tweaking around for a few hours, I managed to get it working


use HTML::HTML5::Parser;

my $user_agent;
$user_agent = LWP::UserAgent->new;
$user_agent("HTML::HTML5::Parser/".'0.110'." ");
$user_agent->parse_head(0);

my $parser = HTML::HTML5::Parser->new;
my $xml = $parser->parse_html_file($URL, {
    ignore_http_response_code => 1,
    user_agent => $user_agent,
    }
);

my $nodes = $xml->findnodes('//*[local-name()="title"]');
say $nodes->get_node(1)->textContent;
[download]

I'm proud I did it, but I don't like it to remove some sort of security checks from the LWP::UserAgent, but somehow, it was nescecary for this website

Question: does it conflict with a HTTP-301 - moved permanently status?

Comment on Re^2: extracting data from HTML Select or Download Code

Replies are listed 'Best First'.
Re^3: extracting data from HTML by bitingduck (Deacon) on Jun 04, 2012 at 04:24 UTC
but ofcourse my test website had to come back with an error One tip for developing scrapers: it's both convenient for you and polite to the site you're scraping to save a local copy that you can hammer at all you want without bothering their server. If you're scraping a lot of pages and doing a lot of tweaking on your code, you have the potential of really hammering someone's server. Once your extractor works, then you can put back the Mechanize calls to the site, which are probably not the hard part In the example I gave upthread, it would have been ok for me to hammer the site, but I ended up cloning it with wget and running it locally. Update: You might also want to see if the site you're scraping has an API that hands you structured data. I recently had to pull down the links for about 140 books from the Apple site, and they have a nice API that lets you search by ISBN. Amazon also tends to have an API for a lot of things. Other sites often do as well if you dig through the fine print at the bottom of the page.	[reply]
Re^4: extracting data from HTML by Jurassic Monk (Acolyte) on Jun 04, 2012 at 18:01 UTC
This whole thing of extracting data from a HTML source is about populating a web-shop. My plan was to harvest as much data as needed and do some 'magick' with it and then use Rpc-XML to update the Magento Database. Guess none of the websites will be friendly in allowing me access to their source, mainly because they do not own their data, they license a web-shop and the data is being provided by another party. is this theft ? - don't answer not every html-source has the same underlying database and some web-sites do provide additional, meaningfull data, that the 'big-player' does not have. So yes, I will make different scrapers for each and every web-site, and even for different product types. The whole process will roughly be something like the following: Enter a product ID get html and save a coache proces data on disk and create source.xml do something meaningful with the sources ask user for confirmation or missing information where needed do something meaningful with the sources and save productinfo.xml take the product.xml and turn it into a magento.xml with XSLT feed the magento database I did a nice job with dirty programming, but the moment I encountered the cp1252 rubbish in my (suposedly) iso-8859-1 I gave up and wanted to start from scratch again, using proper XML modules, no longer relying on XML::Simple, discoverd XSLT that would help me out with processing the different sources and translating them from one (general) data-model to the magento model	[reply]