Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Re: Parsing HTML/XML with Regular Expressions

by Grimy (Pilgrim)
on Oct 17, 2017 at 16:32 UTC ( [id://1201514]=note: print w/replies, xml ) Need Help??


in reply to Parsing HTML/XML with Regular Expressions

Obligatory Zalgo-summoning solution:
#!/usr/bin/perl -p0 s/<!([^<>]|<(?1)*>)*>//gs; s/<(?!div\b[^>]*\bclass\s*=\s*(['"])data\1)([^<>]|<(?2)*>)*>//gs; s/.*?<(?:[^'"]|(['"]).*?\1)*?\bid\s*=\s*(['"])(.*?)\2.*?>([^<]*)/$3=$4 +, /gs; s/&#(\w+);/chr $1/ge; s/[^\w=, ]|, $|(.)\1\1//g;
To make it harder on regexes, I suggest:
  • throwing unbalanced [<>'"] inside CDATA sections / comments / attributes (especially class='data"')
  • using names that can be confused with the interesting ones: <divx, aclass="data", …
  • using XML namespaces liberally
  • using external entities

Replies are listed 'Best First'.
Re^2: Parsing HTML/XML with Regular Expressions
by haukex (Archbishop) on Oct 18, 2017 at 21:11 UTC

    Impressive, thank you! As previously threatened, and as per your comments, some notes on trying to break the regex solution ;-)

    using names that can be confused with the interesting ones: <divx, aclass="data", ...

    Good point, but without some trickery those would no longer validate properly as XHTML either.

    using XML namespaces liberally

    Indeed, I tested this and it does cause trouble: Unsurprisingly the regex and HTML parsers can't handle it, but a little more surprising is that Mojo::DOM ignores namespaces and therefore fails with the following, and that also XML::Twig has trouble with namespaces, or at least I haven't found the right options yet. Only the XML::LibXML and XML::XSH2 solutions handle this correctly:

    <html xmlns:foo="http://www.w3.org/1999/xhtml" xmlns:bar="http://www.perlmonks.com" ... <foo:div class="data" id="Zero" /> <bar:div class="data" id="Hi">there</bar:div>

    (Update: Hmm, even the W3C Validator is having trouble with the namespaces...)

    using external entities

    As noted here, even some XML parsers seem to have trouble loading all the external entities. But even entities declared within the document should make life difficult for regexes:

    <!ENTITY atad "data"> ... <div class="&atad;" id="Zero" />

    Only the XML::LibXML and XML::Twig solutions handle that correctly, everything else (including XML::XSH2) fails.

    Looks like XML::LibXML <update> and XML::XSH2 </update> are the only ones left standing in this torture test so far! :-)

    And one more thing: currently entities with hex values like &#xA0; aren't supported by the regex (although that's not too difficult to fix).

    Updated since the issue with XML::XSH2 was worked out further down in this thread.

      XML::XSH2 is just a wrapper around XML::LibXML. I'd be surprised if it didn't work the same. And indeed, the following doesn't print the id of the div that uses the &atad; class:
      #!/usr/bin/perl use warnings; use strict; use feature qw{ say }; use XML::LibXML; my $dom = 'XML::LibXML'->load_xml(location => '1.xml', load_ext_dtd => + 0); my $xpc = 'XML::LibXML::XPathContext'->new; $xpc->registerNs(xh => 'http://www.w3.org/1999/xhtml'); for my $div ($xpc->findnodes('//xh:div[@class="data"]', $dom)) { print $div->{id}, "\n" }

      Interestingly, at the same time the following shows the classes of all the divs as data:

      for my $div ($xpc->findnodes('//xh:div', $dom)) { print join ' ', @{ $div }{qw{ id class }}, "\n" }

      Bugreport anyone?

      ($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord }map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,

        Could it just be a version issue? The code you posted works fine for me:

        use warnings; use strict; use XML::LibXML; my $XML = <<'_END_XML_'; <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE html [ <!ENTITY atad "data"> ] > <html xmlns="http://www.w3.org/1999/xhtml"> <div class="&atad;" id="Hello" /> <div class="&atad;" id="World" /> </html> _END_XML_ print $XML::LibXML::VERSION, " ", XML::LibXML::LIBXML_DOTTED_VERSION, " ", XML::LibXML::LIBXML_VERSION, " ", XML::LibXML::LIBXML_RUNTIME_VERSION, "\n"; my $dom = XML::LibXML->load_xml(string=>$XML); my $xpc = XML::LibXML::XPathContext->new; $xpc->registerNs(xh => 'http://www.w3.org/1999/xhtml'); for my $div ($xpc->findnodes('//xh:div[@class="data"]', $dom)) { print "1:", $div->{id}, "\n" } for my $div ($xpc->findnodes('//xh:div', $dom)) { print "2:", $div->{id}, " ", $div->{class}, "\n" } __END__ 2.0129 2.9.1 20901 20901 1:Hello 1:World 2:Hello data 2:World data

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1201514]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others examining the Monastery: (8)
As of 2024-03-28 09:16 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found