I wish to match (in a regular expression sense) data in an HTML file. Consider an application wishing to extract data from some web-database like IMDB, CDDB, Yahoo TV, or Perl Monks. In order to do this one must match a tidbit of html containing the data you seek. The data itself is rarely contained in the tags, but by specifying the tag structure, one could in principle write a matcher to find said tidbit.

For a concrete example, consider the following snippet of HTML:

<table>
<th>
  <td>Title</td>
  <td>Subject</td>
  <td>ISBN</td>
  <td>Publisher</td>
  <td>Synopsis</td>
</th>
<tr>
  <td>Ender's Game</td>
  <td>Science Fiction</td>
  <td>12375856996</td>
  <td>Macmillan</td>
  <td>a good book</td></tr>
</tr>
</table>
[download]

If I wished to extract data from this table I might construct a "matcher" using the following (made-up) syntax:

tagblock <td> following tag <tr> inside {
  tagblock <table> containing {
    tagblock <th> {
      containing regex /Subject/
      AND containing regex /ISBN/
      AND containing regex /Publisher/
    }
  }
}
[download]

This would match <td>Ender's Game</td>, successfully extracting the title of the book. Such a matching scheme is relatively impervious to changes in layout, or even changes in the number of table rows or columns.

Consider a second example to find a doubleclick.net advertisement:

tagblock <table width=468 height=60> containing {
  (tag <img> AND tag <a href =~ /doubleclick.net/>)
  OR tag <a href =~ /flycast.com/>
  OR tag <!--/Begin Ad/-->
}

regex /doubleclick.net/ inside {
  attrib </img|script|i?frame|i?layer/ src>
  OR attrib <a href>
} inside tag <a|img|script|i?frame|i?layer>
[download]

These examples are verbose, and are examples of something that doesn't exist, but I hope the syntax is demonstrative of what I'm trying to do. This could be done using HTML::Parser or HTML::TokeParser and a good bit of work, but it would be very slow (consider the second example: finding doubleclick.net with a regex is much faster than examining every tag with HTML::Parser), and it would be fragile with respect to changes in the HTML document's structure, layout, and content. Of course, once the grammer is written and parsed (Parse::RecDescent), one could implement the matcher using HTML::Parser. By creating a grammar for doing pattern matching on HTML, in a manner that obeys document structure, one could have fast matching with a relatively intuitive matching language. Trying to obey document structure allows one to then modify the matched tidbit without destroying document structure by leaving unclosed tags.

The question is: has this been done? Has anything similar been attempted? Is anyone working on such a thing? Does anyone want to work on such a thing? Do other people besides me have a use for such a thing?

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.