Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I'm new to Perl.

OBJECTIVE

I want to automate editing of textual parts of HTML files.

APPROACH

At the abstract level, an HTML document is a tree of nodes. I want to traverse the tree and edit its text nodes.

PROBLEM

Unfortunately, I can't give a simple demonstration because this website translates HTML tags and special characters in posts. I have to verbalize an example.

Suppose in an HTML file to be edited I have

(a) “ampersand” + “lt;” string to represent a left angle bracket “<”, and

(b) “--” string.

Problem 1

After I've built the tree using HTML::TreeBuilder, the “ampersand” + “lt;” string is printed as an ordinary “<” sign.

This is not acceptable.

Problem 2

After I've built the tree using HTML::TreeBuilder, I edit “--” to “M dash”, i.e., “ampersand” + “mdash;”. Printed output is “ampersand” + “amp;” + “mdash;”

This is not acceptable either.

QUESTION

Could you, pray, help me get correct output.

  • Comment on processing text nodes using HTML::Element

Replies are listed 'Best First'.
Re: processing text nodes using HTML::Element
by hossman (Prior) on Mar 08, 2003 at 01:32 UTC

    In both of your problems, the situation is most likely that the method you are using to pull the text out of the nodes (or put text into a node) doesn't do the escaping/unescaping/ignoring of entities that you are expecting, possibly because you aren't specifing the options that you want (probably because you don't kow about those options).

    HTML::TreeBuilder is a subclass of HTML::Element ... take a look at the "as_HTML" method in HTML::Element, and the optional "$entities" parameter to see how to control what it escapes.

    I'll leave it a an excercise for the reader(s) to find the corrisponding way of controlling how text you add to nodes gets escaped.

[reply]

      Hmmm. That's not exactly what I expected for an answer but I will be civil and thank you.

      Let's say I'd done my homework studying the documentation and experimenting before I posted my question. Let's say I've done more reading and experimenting after my post.

      I'm not consistently getting output to my liking.

      Now, if you think you know the right way, would you be less facetious and more specific ?

      Mucho gracias

[reply]
        I'm not consistently getting output to my liking.

        Well, since you haven't posted any code, or even mentioned which methods you are currently using, there's not a lot many people here can do to offer you advice -- other then suggest the methods/options we think you might want to use, and provide links to their documentation. Which I did

[reply]
Re: processing text nodes using HTML::Element
by demerphq (Chancellor) on Mar 08, 2003 at 12:59 UTC

    Unfortunately, I can't give a simple demonstration because this website translates HTML tags and special characters in posts.

    Didn't you notice lots of posts with proper code and formatting? Surely it occured to you that you too could do the same? :-) Use code tags. <code></code>

    Read the Writeup Formatting Tips. Show us the code you having problems with. Preferably the smallest piece of code that reproduces the behaviour you want changed. Then we can maybe post a suggestion.


    ---
    demerphq


[reply]
[d/l]

      Didn't you notice lots of posts with proper code and formatting? Surely it occured to you that you too could do the same? :-) Use code tags. <code></code>

      Signor demerphq,

      Thank you for your reply.

      Yes, I had. But for some reason on previewing I didn't like what I saw.

      Cheers

[reply]
[d/l]
Re: processing text nodes using HTML::Element
by bbfu (Curate) on Mar 09, 2003 at 01:07 UTC

    You need to look at the '~literal' pseudo-element of HTML::Element.

    If I understand your requirements correctly, the following code should do what you want. 

    #!/usr/bin/perl
use warnings;
use strict;
my $html = <<'EOHTML';
<html>
 <body>
  <p>The relationship can be expressed by the following equation:<br>
  <blockquote>
    y &lt;= (7x<sup>3</sup> + 3x<sup>2</sup>)/((x - 3)(x - 5)
  </blockquote>

  <p>Of course x != 3 &amp; x != 5  --  That goes without saying.
 </body>
</html>
EOHTML

use HTML::TreeBuilder;

my $tree = HTML::TreeBuilder->new_from_content($html);

# Look for double-dashes in the text,
# and change them to &mdash;es.
$tree->look_down(sub {
  my $element = shift;
  my @content = $element->content_list();

  @content = map {
    if(ref) {
      $_       # Skip non-text children
    } else {

      # Break up the text child into pieces on the --'s
      my @texts = split /(--)/, $_;
      # Replaces those --'s with a ~literal
      # pseudo-element containing an &mdash;
      @texts = map {
        $_ eq '--'
          ? HTML::Element->new('~literal', text => '&mdash;')
          : $_
      } @texts;

      @texts;
    }
  } @content;

  # Replace the old content with the modified content
  $element->splice_content(0, scalar $element->content_list, @content)
+;

  return 0;
});

print $tree->as_HTML;

    [download]

    Update: The (formatted) output of the above is this: 

    <html>
 <head>
 </head>
 <body>
  <p>The relationship can be expressed by the following equation:<br>
   <blockquote>
    y &lt;= (7x<sup>3</sup> + 3x<sup>2</sup>)/((x - 3)(x - 5)
   </blockquote>
  <p>Of course x != 3 &amp; x != 5 &mdash; That goes without saying.
 </body>
</html>

    [download]

    bbfu
    Black flowers blossum
    Fearless on my breath

[reply]
[d/l]
[select]

      bbfu,

      Thank you for your reply.

      I perused your code, not running it. I trust it works. In fact, at one point I'd thought I'd have to do it "your" way.

      Let me backtrack and get certain facts straight. From my reading of the documentation I understand there are two ways of controlling how the printed output of a document tree will look.

      (a) The $entities parameter passed to the as_HTML method.
      (b) Using the '~literal' pseudo-element.

      Am I right or missing something ?

      Let me keep on ruminating. In my situation, I've dozens of HTML files whose contents I know nothing about, and should not have to care to know about. By contents I mean text the user can see rendered in the browser—not markup.

      My program reflects the following process:

      (a) read an HTML input file;
      (b) construct a document tree;
      (c) repair text nodes, traversing the tree;
      (d) print the tree as an HTML file.

      Suppose my input HTML file contains the following string:

      &mdash;&nbsp;&copy;

      When my program goes through steps (a), (b) and (d), omitting the step (c), the string in the output file is DIFFERENT from that in the input file.

      Before I go any further in this conversation, let me pause here and ask this question:

      Do you know how to get from the document tree exactly the same text as “went into” the tree ?

      Thank you for your time.

      Cheers

      P.S.

      I'm using ActiveState's Perl 5.6.1 on Win2K.

[reply]
[d/l]

        Technically, I'm not sure that the '~literal' pseudo-element actually has any effect on how things are printed out (though that's how it's explained in the docs). It seems more sensical to me that it controls whether the text inserted is escaped or not. *shrug* But that's just implementation details. :)

        Honestly, I don't really think there's a way to get HTML::TreeBuilder to print out the original HTML exactly the same. You can get pretty close by calling $tree->no_space_compacting(1) before parsing the HTML, but it doesn't seem to be 100% the same.

        If all you want to do is make some minor changes (eg, changing '--' to '&mdash;') to the text parts, keeping the rest of the HTML exactly the same, you're probably best off switching to HTML::Parser (which HTML::TreeBuilder is based on). Code follows... 

        #!/usr/bin/perl
use warnings;
use strict;

our $html = <<'EOHTML';
<html>
 <body>
  <p>The relationship can be expressed by the following equation:<br>
  <blockquote>
    y &lt;= (7x<sup>3</sup> + 3x<sup>2</sup>)/((x - 3)(x - 5)
  </blockquote>
  <p>Of course x != 3 &amp; x != 5  --  That goes without saying.
  <p>:)
 </body>
</html>
EOHTML

use HTML::Parser;

my $parser = HTML::Parser->new(
  handlers => {
    default => [sub { print @_ }, 'text'], # print out HTML tags unmod
+ified
    text    => [\&process_text, 'text'],
  },
);
$parser->parse($html);

sub process_text {
  $_ = shift;

  # Make any changes to the text here...

  s/--/&mdash;/g;
  s/:\)/<img src="smiley.gif">/g;

  # and so forth.

  # Now print out the modified text.
  print;
}

        [download]

        Output: 

        <html>
 <body>
  <p>The relationship can be expressed by the following equation:<br>
  <blockquote>
    y &lt;= (7x<sup>3</sup> + 3x<sup>2</sup>)/((x - 3)(x - 5)
  </blockquote>
  <p>Of course x != 3 &amp; x != 5  &mdash;  That goes without saying.
  <p><img src="smiley.gif">
 </body>
</html>

        [download]

        bbfu
        Black flowers blossum
        Fearless on my breath

[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom