in reply to Re^2: extracting data from HTML
in thread extracting data from HTML

Yes, it returns plain text because the textContent method is documented as:

this function returns the content of all text nodes in the descendants of the given node as specified in DOM. (perldoc XML::LibXML::Node)

Data::Dumper won't be much use with XML::LibXML. Nodes are all just numeric pointers to structures at the other side of the XS boundary (i.e. C structures). There is XML::LibXML::Debugging which allows, e.g.

  print Dumper( $xml->toDebuggingHash );

[download]
perl -E'sub Monkey::do{say$_,for@_,do{($monkey=[caller(0)]->[3])=~s{::}{ }and$monkey}}"Monkey say"->Monkey::do'

Replies are listed 'Best First'.
Re^4: extracting data from HTML
by Jurassic Monk (Acolyte) on Jun 05, 2012 at 20:56 UTC

    forgive me my brethern but it looks I did bite off more then I could chew and again I endup with bits I can not put together

    the example of
    my $nodes = $xml->findnodes('//*[local-name()="title"]')
    wasn't to much to understand, altohug quite surprised with the construction of the xpath-expression; I would had expected something more easy like
    my $nodes = $xml->findnodes('//html/head/title]')
    But ofcourse, it wouldn't be me if I would get it wrong again

    With HTML::TreeBuilder::XPath it did work, even things like giving me all the table rows from a specific path and dump as text with 

    my @stuff =
    $tree->findvalues( '//td[@class="BTrow"]/table/tr/td/table/tr');
print Dumper(\@stuff);

    [download]

    Trying that with HTML::HTML5::Parser only resulted in undefined results

    It looks to me I'm missing some bit

    please toby, and others as well, what am I doing wrong here, it can't be the xpath syntax, is it?

    thank you all for your enlighting words and inspiration

[reply]
[d/l]
[select]

      The problem with this:

      $xml->findnodes('//html/head/title')

      [download]

      ... is that none of the names in the path include a namespace. (X)HTML elements are always namespaced. Hence my rather awkward...

      $xml->findnodes('//*[local-name()="title"]')

      [download]

      i.e. "select all elements where the local part of the element's name is 'title'"

      Another solution (arguably a lot more readable) would be to forcibly bind the XHTML namespace to a prefix:

      $xml->documentElement->setNamespace(
  'http://www.w3.org/1999/xhtml' => 'xh',
);

      [download]

      And then you can freely use that prefix in XPaths.

      $xml->findnodes('//xh:html/xh:head/xh:title')

      [download]

      This specific problem is mentioned in the XML::LibXML::Node documentation - look for the "NOTE ON NAMESPACES AND XPATH" in the documentation for the findnodes method.

      perl -E'sub Monkey::do{say$_,for@_,do{($monkey=[caller(0)]->[3])=~s{::}{ }and$monkey}}"Monkey say"->Monkey::do'
[reply]
[d/l]
[select]

        that is pretty awkward!

        strangely enough it seems that the other modules do not bother about the namespaces.

        The reasoning behind the namespaces and that those thus need to be included in the XPath is clear. Parsing the badly formed HTML into a XML-structure should ofcourse lead to proper XHTML, I do see that. Hence nsURI does use http://www.w3.org/1999/xhtml.

        I made a dump of the 'title' node: 

                  'nsuri' => 'http://www.w3.org/1999/xhtml',
          'suffix' => 'title',
          'qname' => 'title',
          'children' => [
                          'extracting data from HTML'
                        ],
          'type' => 'Element',
          'attributes' => [],
          'prefix' => undef

        [download]

        As the webpage of the referenced perlmonks node is just plain html with <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">, I will asume, somewhere the http://www.w3.org/1999/xhtml must have slipped in. It would be nice to override that - or, it would be nice to just remove it from each and every node in the XML-Document.

        Maybe there is a way to bypass those default namespaces, or do some quirks on the XPath expression, before applying it to the document

        All in all... yes I do desire to be fully knowlageable off all ins and outs that come with XML, XML-Schema, XPath, XSLT and more...

        But I don't apreciate this awkward namespace trouble

        Brethern monks, is there a way to get around this, other than inseting my own prefix?


        N.B. almost every example off XPath expressions, this is overlooked and simplyfied, which is a nasty bummer when you want to work with it

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom