Re: Re (tilly) 2: HTML Matching

This matches "regular" HTML tags -- the part that matches the element may need to be changed slightly, but other than that, it matches: <ELEMENT ( ATTR ( = VALUE )? )* >.

my $open = qr{
  < [a-zA-Z][a-zA-Z0-9]*
  (?:
    \s+ \w+
    (?: \s* = \s*
      (?: "[^"]*" | '[^']*' | [^\s>]* )
    )?
  )*
  \s*
  >
}x;
[download]

The closing tags are far simpler:

my $close = qr{
  < / \s* [a-zA-Z][a-zA-Z0-9]* \s* >
}x;
[download]

Comments are slightly trickier:

# the following are comments:
# <!-- ab -- cd -->   <!-- ab -->    <!---->
# <!-- ab -- cd -- >  <!-- ab -- >   <!---- >

my $comment = qr{
  <!--              # <!--
  [^-]*             # 0 or more non -'s
  (?:               
    (?! -- \s* > )    # that's not --, space, then >
    -                 # a -
    [^-]*             # 0 or more non -'s
  )*                # 0 or more times
  -- \s* >          # --, space, then >
}x;
[download]

The DTD tag is more difficult. There are specific classes of DTD tags (see the specs). So right onw I don't have a regex to handle them. But combining the other three regexes:

while ($HTML =~ /\G($open|$close|$comment|[^<]+)/g) {
  # do something with $1
}
[download]

Now, using this to create a tree structure of an HTML file shouldn't be too complicated, especially if we use a nice trick like:

# requires the (?{...}) structure
use re 'eval';

while ($HTML =~ m{
  \G
  (
    $open    (?{ $STATE = 'open'    }) |
    $close   (?{ $STATE = 'close'   }) |
    $comment (?{ $STATE = 'comment' }) |
    [^<]+    (?{ $STATE = 'TEXT'    })
  )
}xg) {
  # do something with $1 and $STATE
}
[download]

And you can modify $open and $close to keep track of the element name by putting parens in there.

It's a matter of thoroughness.

japhy -- Perl and Regex Hacker

Comment on Re: Re (tilly) 2: HTML Matching Select or Download Code

Replies are listed 'Best First'.
Re (tilly) 4: HTML Matching by tilly (Archbishop) on Nov 19, 2000 at 03:48 UTC
I love that tree comment. It is a classic. :-) The tree structure involves such details as the fact that tags must balance and can involve nested tables. Matching nesting with a single regular expression is impossible, you cannot even construct an RE that tests whether parentheses balance! Another major detail you are skipping over is the fact that there are a lot of constructs which are invalid HTML but the browsers attempt to work around. In a real environment if someone can find some broken construct that escapes your match but gets interpreted the right way by some browser, that is a major oops. Enjoy your RE, but I won't trust it. At some point you need to stop thinking about this as a matching problem which RE's are designed for) and start thinking of this as a parsing problem (which REs are a useful tool for but are only a piece of the puzzle). As soon as you start saying that you are parsing the document, it becomes trivial to be sure that you are not missing anything and you will guarantee that you put out a valid document which cannot cause confusion. Random thought: Have you thought about how you will catch someone who just puts `<--` somewhere? What if someone designs a tag which you strip out - and in the process turn something you skipped because it was an invalid tag into a valid one? How many more of these can you think of, and how are you going to convince me that you have caught them all? A non-RE solution can easily - demonstrably - result in output that is absolutely guaranteed to be safe. I stand by my doubt that an RE solution will ever achieve that status.	[reply]

Replies are listed 'Best First'.

Re (tilly) 4: HTML Matching
by tilly (Archbishop) on Nov 19, 2000 at 03:48 UTC

The tree structure involves such details as the fact that tags must balance and can involve nested tables. Matching nesting with a single regular expression is impossible, you cannot even construct an RE that tests whether parentheses balance!

Another major detail you are skipping over is the fact that there are a lot of constructs which are invalid HTML but the browsers attempt to work around. In a real environment if someone can find some broken construct that escapes your match but gets interpreted the right way by some browser, that is a major oops.

Enjoy your RE, but I won't trust it. At some point you need to stop thinking about this as a matching problem which RE's are designed for) and start thinking of this as a parsing problem (which REs are a useful tool for but are only a piece of the puzzle). As soon as you start saying that you are parsing the document, it becomes trivial to be sure that you are not missing anything and you will guarantee that you put out a valid document which cannot cause confusion.

Random thought: Have you thought about how you will catch someone who just puts <-- somewhere? What if someone designs a tag which you strip out - and in the process turn something you skipped because it was an invalid tag into a valid one? How many more of these can you think of, and how are you going to convince me that you have caught them all?

A non-RE solution can easily - demonstrably - result in output that is absolutely guaranteed to be safe. I stand by my doubt that an RE solution will ever achieve that status.

[reply]