Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??

If, as it sounds from your post, you seeking to code your algorithm to be entirely independent of the pages and their content that you are intending to parse, you are on a hiding to nothing.

To demonstrate the difficulty, first construct a short set of data that you might hope to be able to extract. Say:

thingy red 10.00 "new fresh thingy" dobrey blue 1.99 "antique doobree" whatsit transparent 16.49/yard indescribable

Now consider all the hundreds of different ways you could wrap that up in html in order to display it.

Then consider how many variations you could contrive of each of those hundreds of ways by adding a little javascript into the mix.

Then consider the effects of adding in images; filler; adverts; links to customer reviews; pagination controls; 'you might also like to consider' and 'other customers also brought" lists; and all the other irrelevances and annoyances that you routinely encounter on websites.

You end up with trying to consider how to extract those same 12 pieces of data from 100s of thousands of different formats, before even considering the possibilities of different languages or deliberate obfuscation to prevent scraping. You could spend months attempting to write such a generic parser only to be foiled when they revamp their websites.

It would be much better to tailor simple front-end scrappers to each of your specific target pages, and only get generic once you have extracted data you require. That way, when one page format changes, or a new page format needs to be handled, you are only faced with modifying (or writing anew) a small front-end script, not trying to adapt your entire parser to the new without breaking the existing.


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

In reply to Re: Machine learning pattern matching... by BrowserUk
in thread Machine learning pattern matching... by cLive ;-)

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others about the Monastery: (5)
As of 2024-03-28 14:10 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found