Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??

HTML::Tree is your friend. I used it very extensive in converting old pages to new formats, pulling out content, and even XML parsing.

As for identifying various structures, the easiest approach I found was the following:

Presuming you have like 10,000 pages of somehow similar structure but have suspicions there might be some different. Build the parse tree for each page and eliminate anything which isn't structural from the tree. Dump the html with no spaces, no comments to a string and make a hash out of that string. For each hash in this stage keep a list of files that matched it and the "blueprint" in some structures of your choice (mine are files for example)... After this step you'll get some classes of pages, each matching one blueprint.

After you get the structures based on the blocks on the pages, repeat the same procedure for each block and contained information.

The case in which this method got the best results was extracting information out of about 18,000 pages which contained something like 3 tables of key-values pairs, some free form content (descriptions, reviews, comments), images and some other blocks. The layout was a mess, part of the pages being initially generated through 3-4 different scripts and then updated manually over a few years with anything from text editors to *cough* FrontPage... the result was pretty impressive... in just about a day (actually a night), over 99% of the pages were in a nice database.


In reply to Re: Validating HTML structures by b374
in thread Validating HTML structures by wfsp

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others contemplating the Monastery: (3)
As of 2024-04-25 23:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found