Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I've got loads of files to process, typical content:
<div class="spec_section"> <div class="spec_row"> <div class="spec_row_header orange_text_1"> Engine </div> </div> <div class="spec_row"> <div class="spec_row_left"> Type </div> <div class="spec_row_right"> Diesel </div> </div> <div class="spec_row"> <div class="spec_row_left"> Builder </div> <div class="spec_row_right"> MTU </div> </div> <div class="spec_row"> <div class="spec_row_left"> Model </div> <div class="spec_row_right"> 2x 16V396TB94 </div> </div> <div class="spec_row"> <div class="spec_row_left"> Power </div> <div class="spec_row_right"> 2561kw / 3480hp </div> </div> <div class="spec_row"> <div class="spec_row_left"> Total Power </div> <div class="spec_row_right"> 5121kw / 6960hp </div> </div> <div class="spec_row"> <div class="spec_row_left"> Engine Propulsion </div> <div class="spec_row_right"> Twin Screws </div> </div> </div>

What's the best way to convert this to XML, eg:

<engine> <type>Diesel</type> <builder>MTU</builder> etc. </engine>
Thanks!

Replies are listed 'Best First'.
Re: module for HTML XML conversion
by Your Mother (Archbishop) on Mar 27, 2010 at 17:18 UTC

    (update: the later example, Re: module for HTML XML conversion, is much better than this one.)

    This is pretty specialized and inlines more than it should. You have to write a few lines of code for every part of your spec. So it is not the best if you have a need for generalized transforms but if you have an exact transform to do that won't change much, this can be great. XML::LibXML.

    use warnings; use strict; use XML::LibXML; my $parser = XML::LibXML->new(); $parser->keep_blanks(0); # $parser->recover_silently(1); # <-- for bad HTML my $doc = $parser->parse_fh(\*DATA); # If you have well balanced snipp +et like below # my $doc = $parser->parse_html_fh(\*DATA); # If you have HTML. my $xml = XML::LibXML::Document->new(); $xml->setDocumentElement( $xml->createElement("engines") ); for my $spec_row ( $doc->findnodes('//div[@class="spec_row"]') ) { my $engine = $xml->createElement("engine"); $xml->getDocumentElement->appendChild($engine); my $type = $xml->createElement("type"); ( my $text = $spec_row->firstChild->textContent ) =~ s/\A\s+|\s+\z +//g; $type->appendChild( $xml->createTextNode($text) ); $engine->appendChild( $type ); my $builder = $xml->createElement("builder"); ( $text = $spec_row->lastChild->textContent ) =~ s/\A\s+|\s+\z//g; $builder->appendChild( $xml->createTextNode($text) ); $engine->appendChild( $builder ); } print $xml->serialize(1); __DATA__ <div class="spec_section"> <div class="spec_row"> <div class="spec_row_header orange_text_1"> Engine </div> </div> <div class="spec_row"> <div class="spec_row_left"> Type </div> <div class="spec_row_right"> Diesel </div> </div> <div class="spec_row"> <div class="spec_row_left"> Builder </div> <div class="spec_row_right"> MTU </div> </div> <div class="spec_row"> <div class="spec_row_left"> Model </div> <div class="spec_row_right"> 2x 16V396TB94 </div> </div> <div class="spec_row"> <div class="spec_row_left"> Power </div> <div class="spec_row_right"> 2561kw / 3480hp </div> </div> <div class="spec_row"> <div class="spec_row_left"> Total Power </div> <div class="spec_row_right"> 5121kw / 6960hp </div> </div> <div class="spec_row"> <div class="spec_row_left"> Engine Propulsion </div> <div class="spec_row_right"> Twin Screws </div> </div> </div>

      my $doc = $parser->parse_fh(\*DATA); # If you have well balanced snippet like below

      Is there any reason to not use parse_html_fh to parse HTML? All of the following are well balanced (or impossible to balance) but invalid XML:

      <option selected ...>...</option> <font color=red>...</font> <br> <img ...> <meta ...> &nbsp; (without XML bits to define it)

      Encoding is also handled differently.

        No, I think it's a good idea. But the sample input was well balanced so I used it and included the other in (as you point out, the likely) case it wouldn't fly.

        The sample also fails to specially account for the first <div/> which contains the "column" names. We could probably do something automatic with that instead of hard coding the column parsing. The OP might have thought of that already. I didn't until looking it over again now. If I have a few minutes I'll revisit it with that approach.

      That's very kind of you to do that - I can definitely work on that approach. Much appreciated...
Re: module for HTML XML conversion
by ikegami (Patriarch) on Mar 27, 2010 at 17:09 UTC

    From what I've seen, XML::Twig is great at transforming XML. If you could transform your HTML to XHTML somehow, then you could use XML::Twig to transform the XHTML into the XML subformat you desire.

      HTML::TreeBuilder + ->as_XML
Re: module for HTML XML conversion
by jonnyfolk (Vicar) on Mar 27, 2010 at 16:43 UTC
    HTML::Tidy?
Re: module for HTML XML conversion
by Your Mother (Archbishop) on Mar 29, 2010 at 01:43 UTC

    I think my first example was not getting very close to what you actually want. Try this instead. It's also a bit weird, tricky, and not robust but the output seems right this time and it's semi-automatic for finding key, value pairs.

    use warnings; use strict; use XML::LibXML; sub snipName { s/\s+//g and $_ = lcfirst($_) for @_ }; sub snipValue { s/\A\s+|\s+\z//g for @_ }; my $parser = XML::LibXML->new(); $parser->keep_blanks(0); # $parser->recover_silently(1); # Might need, might not. my $doc = $parser->parse_html_fh(\*DATA); my $xml = XML::LibXML::Document->new(); my $root_name = [ $doc->findnodes('//div[@class="spec_section"]//div[c +ontains(@class,"spec_row_header")]') ]->[0]->textContent; snipName($root_name); $xml->setDocumentElement( $xml->createElement($root_name) ); for my $col ( $doc->findnodes('//div[@class="spec_row_left"]') ) { snipName( my $name = $col->textContent ); my $tag = $xml->createElement($name); my $value = [ $col->parentNode->findnodes('div[@class="spec_row_ri +ght"]') ]->[0]->textContent; snipValue( $value ); $tag->appendChild( $xml->createTextNode( $value ) ); $xml->getDocumentElement->appendChild($tag); } print $xml->serialize(1); __DATA__ <div class="spec_section"> <div class="spec_row"> <div class="spec_row_header orange_text_1"> Engine </div> </div> <div class="spec_row"> <div class="spec_row_left"> Type </div> <div class="spec_row_right"> Diesel </div> </div> <div class="spec_row"> <div class="spec_row_left"> Builder </div> <div class="spec_row_right"> MTU </div> </div> <div class="spec_row"> <div class="spec_row_left"> Model </div> <div class="spec_row_right"> 2x 16V396TB94 </div> </div> <div class="spec_row"> <div class="spec_row_left"> Power </div> <div class="spec_row_right"> 2561kw / 3480hp </div> </div> <div class="spec_row"> <div class="spec_row_left"> Total Power </div> <div class="spec_row_right"> 5121kw / 6960hp </div> </div> <div class="spec_row"> <div class="spec_row_left"> Engine Propulsion </div> <div class="spec_row_right"> Twin Screws </div> </div> </div>

    Which yields-

    <?xml version="1.0"?> <engine> <type>Diesel</type> <builder>MTU</builder> <model>2x 16V396TB94</model> <power>2561kw / 3480hp</power> <totalPower>5121kw / 6960hp</totalPower> <enginePropulsion>Twin Screws</enginePropulsion> </engine>
      FWIW I am doing this with HTML Tidy ... via the shell http://www.ibm.com/developerworks/library/x-tiptidy.html
Re: module for HTML XML conversion
by Anonymous Monk on Mar 27, 2010 at 15:02 UTC
    What's the best way to convert this to XML

    Javascript DOM