in reply to Fetching meta tag info

I think it is worth adding to this thread a quick explanation on why regular expressions are a very risky way to process HTML (as well as XML).

HTML is a very complex standard, the rules about what is a tag, how they can be ommited and infered, about whitespaces etc... are really complex.

Here are a list of proper meta tags that "naive" or even quite advanced regular expressions would fail to match (all are correctly parsed by HTML::TokeParser and I guess by HTML::TokeParser::Simple):

<meta  NAME="HTML.author" CONTENT="Joe Smith">    <!-- 2 spaces before
+ NAME -->
<meta NAME=HTML.author CONTENT="Joe Smith">       <!-- no " around the
+ value of NAME -->
<meta NAME="HTML.author" CONTENT='Joe Smith'>     <!-- you can use ' i
+nstead of " -->
<meta CONTENT="Joe Smith" NAME="HTML.author">     <!-- the order of at
+tributes is not significant -->
<meta NAME="HTML.author"
      CONTENT="Joe Smith">                        <!-- tags can span a
+ccross several lines-->
<meta NAME="HTML.author" CONTENT="Joe Smith" />   <!-- XHTML style -->

[download]

I am sure other examples could be found.

And of course you might grab things that look like a meta tag but are not, like:

<!-- <meta NAME="HTML.author" CONTENT="Joe Smith"> --> <!-- yep, a comment -->

Replies are listed 'Best First'.
Re: Re: Fetching meta tag info
by dingus (Friar) on Nov 13, 2002 at 08:21 UTC
    I think it is worth adding to this thread a quick explanation on why regular expressions are a very risky way to process HTML (as well as XML).

    <many excellent examples SNIPPed>

    Al the above is absolutely true for arbitrary webpages - as in the problem you repsonded to.

    However one of the more common LWP etc. tasks is to retrieve and search for/extract information from the web front ends of query tools (such as Pubmed or Scirus (or google although google doesn't like LWP)) or other template built webpages (such as this one). This is where the recent micro-optimisation comments apply. In such a case regexes or even simpler index($html, $string) statements are much faster and consume fewer resources. The difference can mean that your script can run OK on an ancient PC versus requiring oodles of processing power/memory.

    Moreover for that kind of operation navigating the complex hashes and hashes of arrays of hashes etc. produced by HTML::TokeParser::* and relatives is probably harder to maintain in the event of layout changes than the regex method. Consider what happens when a TABLE is replaced by a set of positioned DIVs and ULs for example, but all you want is to see if "Fred Bloggs" appears anywhere in the results section.

    Summary: If you want to get a small amount of information from a particular tag or tags from arbitrary HTML then TokeParser is definitely the way to go. On the other hand if you want to extract large amounts of content from a series of similar pages then regexes are not a bad way to do things.

    Dingus

    Enter any 47-digit prime number to continue.
[reply]

In Section Seekers of Perl Wisdom