if I do * then I get back all of the elements from the whole tree

Not true. Or at least I do not get it, and you should not get it either:

perl -E '
    use XML::LibXML;
    my $dom = XML::LibXML->load_xml(string => q{
<library>
  <book attr1="zzz">
    <title>xxxx</title>
    <author>yyyyy</author>
  </book>
  <book attr1="sssss"> ---- </book>
  <book attr1="dddd"> ---- </book>
</library>
    });
    my ($l) = $dom->findnodes("/library");
    say $_->getName for $l->findnodes("*[text()] | text()");'
#text
book
#text
book
#text
book
#text
[download]