Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Re: Validating HTML structures

by b374 (Initiate)
on Oct 31, 2005 at 00:27 UTC ( [id://504092]=note: print w/replies, xml ) Need Help??


in reply to Validating HTML structures

HTML::Tree is your friend. I used it very extensive in converting old pages to new formats, pulling out content, and even XML parsing.

As for identifying various structures, the easiest approach I found was the following:

Presuming you have like 10,000 pages of somehow similar structure but have suspicions there might be some different. Build the parse tree for each page and eliminate anything which isn't structural from the tree. Dump the html with no spaces, no comments to a string and make a hash out of that string. For each hash in this stage keep a list of files that matched it and the "blueprint" in some structures of your choice (mine are files for example)... After this step you'll get some classes of pages, each matching one blueprint.

After you get the structures based on the blocks on the pages, repeat the same procedure for each block and contained information.

The case in which this method got the best results was extracting information out of about 18,000 pages which contained something like 3 tables of key-values pairs, some free form content (descriptions, reviews, comments), images and some other blocks. The layout was a mess, part of the pages being initially generated through 3-4 different scripts and then updated manually over a few years with anything from text editors to *cough* FrontPage... the result was pretty impressive... in just about a day (actually a night), over 99% of the pages were in a nice database.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://504092]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others exploiting the Monastery: (7)
As of 2024-04-19 16:35 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found