Re^2: Parsing HTML/XML with Regular Expressions

Impressive, thank you! As previously threatened, and as per your comments, some notes on trying to break the regex solution ;-)

using names that can be confused with the interesting ones: <divx, aclass="data", ...

Good point, but without some trickery those would no longer validate properly as XHTML either.

using XML namespaces liberally

Indeed, I tested this and it does cause trouble: Unsurprisingly the regex and HTML parsers can't handle it, but a little more surprising is that Mojo::DOM ignores namespaces and therefore fails with the following, and that also XML::Twig has trouble with namespaces, or at least I haven't found the right options yet. Only the XML::LibXML and XML::XSH2 solutions handle this correctly:

<html xmlns:foo="http://www.w3.org/1999/xhtml"
      xmlns:bar="http://www.perlmonks.com"
...
<foo:div class="data" id="Zero" />
<bar:div class="data" id="Hi">there</bar:div>
[download]

(Update: Hmm, even the W3C Validator is having trouble with the namespaces...)

using external entities

As noted here, even some XML parsers seem to have trouble loading all the external entities. But even entities declared within the document should make life difficult for regexes:

<!ENTITY atad "data">
...
<div class="&atad;" id="Zero" />
[download]

Only the XML::LibXML and XML::Twig solutions handle that correctly, everything else ~~(including XML::XSH2)~~ fails.

Looks like XML::LibXML <update> and XML::XSH2 </update> are the only ones left standing in this torture test so far! :-)

And one more thing: currently entities with hex values like   aren't supported by the regex (although that's not too difficult to fix).

Updated since the issue with XML::XSH2 was worked out further down in this thread.

Comment on Re^2: Parsing HTML/XML with Regular Expressions Select or Download Code

Replies are listed 'Best First'.
Re^3: Parsing HTML/XML with Regular Expressions by choroba (Cardinal) on Oct 18, 2017 at 21:52 UTC
XML::XSH2 is just a wrapper around XML::LibXML. I'd be surprised if it didn't work the same. And indeed, the following doesn't print the id of the div that uses the `&atad;` class: `#!/usr/bin/perl use warnings; use strict; use feature qw{ say }; use XML::LibXML; my $dom = 'XML::LibXML'->load_xml(location => '1.xml', load_ext_dtd => + 0); my $xpc = 'XML::LibXML::XPathContext'->new; $xpc->registerNs(xh => 'http://www.w3.org/1999/xhtml'); for my $div ($xpc->findnodes('//xh:div[@class="data"]', $dom)) { print $div->{id}, "\n" }` [download] Interestingly, at the same time the following shows the classes of all the divs as `data`: `for my $div ($xpc->findnodes('//xh:div', $dom)) { print join ' ', @{ $div }{qw{ id class }}, "\n" }` [download] Bugreport anyone? ($q=q:Sq=~/;[c](.)(.)/;chr(-\|\|-\|5+lengthSq)`"S\|oS2"`map{chr \|+ord }map{substrSq`S_+\|`\|}3E\|-\|`7**2-3:)=~y+S\|`+$1,++print+eval$q,q,a, [download]	[reply] [d/l] [select]
Re^4: Parsing HTML/XML with Regular Expressions by haukex (Archbishop) on Oct 18, 2017 at 22:05 UTC
Could it just be a version issue? The code you posted works fine for me: use warnings; use strict; use XML::LibXML; my $XML = <<'_END_XML_'; <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE html [ <!ENTITY atad "data"> ] > <html xmlns="http://www.w3.org/1999/xhtml"> <div class="&atad;" id="Hello" /> <div class="&atad;" id="World" /> </html> _END_XML_ print $XML::LibXML::VERSION, " ", XML::LibXML::LIBXML_DOTTED_VERSION, " ", XML::LibXML::LIBXML_VERSION, " ", XML::LibXML::LIBXML_RUNTIME_VERSION, "\n"; my $dom = XML::LibXML->load_xml(string=>$XML); my $xpc = XML::LibXML::XPathContext->new; $xpc->registerNs(xh => 'http://www.w3.org/1999/xhtml'); for my $div ($xpc->findnodes('//xh:div[@class="data"]', $dom)) { print "1:", $div->{id}, "\n" } for my $div ($xpc->findnodes('//xh:div', $dom)) { print "2:", $div->{id}, " ", $div->{class}, "\n" } __END__ 2.0129 2.9.1 20901 20901 1:Hello 1:World 2:Hello data 2:World data [download]	[reply] [d/l]
Re^5: Parsing HTML/XML with Regular Expressions by choroba (Cardinal) on Oct 18, 2017 at 22:13 UTC
I modified your original input: <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"[ <!ENTITY atad 'data'> <!ATTLIST html xmlns:xsi CDATA #FIXED "http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation CDATA #IMPLIED > ]> <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.w3.org/1999/xhtml http://www.w3.org/2002/08/xhtml/xhtml1-strict.xsd"> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" + /> <title>Hello, World</title> <script type="text/javascript"> //<![CDATA[ console.log(' <div class="data" id="Hello">World</div> '); //]]> </script> </head> <body> <div class="data" id="Zero" /> <div class="data" id="One">Monday</div><div class="data" id="Two">Tues +day</div> <div id="Three" class='data'>Wednes<div id="day">day</div></div> <div class="&atad;" id='Four'><b>Thursday</b></div> [download] etc. Four was missing from the output for both the libraries. ($q=q:Sq=~/;[c](.)(.)/;chr(-\|\|-\|5+lengthSq)`"S\|oS2"`map{chr \|+ord }map{substrSq`S_+\|`\|}3E\|-\|`7**2-3:)=~y+S\|`+$1,++print+eval$q,q,a, [download]	[reply] [d/l] [select]
Re^6: Parsing HTML/XML with Regular Expressions (updated) by haukex (Archbishop) on Oct 18, 2017 at 22:54 UTC
Re^7: Parsing HTML/XML with Regular Expressions (updated) by choroba (Cardinal) on Oct 19, 2017 at 10:02 UTC


Do you know where your variables are?
	PerlMonks