In such a case I would use CDATA sections. First wrap the content of the dodgy elements in <[CDATA[ ... ]]>, then you can parse them without problem:

#!/bin/perl -w

use strict;
use XML::Twig;
use Data::Dumper;

# generate a file where the content of rec1 and something is
# stuck in CDATA sections
my $tmp="tmp";
open( TMP, ">$tmp") or die "$0 cannot open $tmp: $!"; 
while( <DATA>)
  { s{<(rec1|something)>}{<$1><![CDATA[}g;
    s{</(rec1|something)>}{]]></$1>}g;
    print TMP $_;
  }
close TMP;

# sorry, I could not help but use XML::Twig for this
my %data;

my $t= XML::Twig->new( 
         twig_handlers => { r => sub { $data{$_->field( 'key')}= 
                                        { rec1      => $_->field( 'rec
+1'), 
                                          something => $_->field( 'som
+ething')
                                        };
                                       $_[0]->purge; # I like to save 
+memory 
                                      }
                           },
                      );
$t->parsefile( $tmp);
print Dumper( %data);


__DATA__
<data>
  <r>
    <key>k1</key>
    <rec1>data</rec1>
    <something>else</something>
  </r>
  <r>
    <key>k2</key>
    <rec1>includes <br> and non UTF-8 chars like é, or nasties like < 
+</rec1>
    <something>else, <p>ugly <i>too<b>isn't</i> it</b> & all</p></some
+thing>
  </r>
</data>
[download]

The only restrictions are: if the "embedded" data contains ]]> then you must break the CDATA section, and the data of course it must not contain </rec1> or </something>.

I am afraid gav^ is right: you can't use XML::Parser, or any XML module for what matters, if your data is not well-formed XML.

Which gives you 3 choices:

• use a custom parser, that deals with the data you actually have, just do not call it XML, believe me it will save you tons of problems in the long run, when you want to use the parser on real XML,
• use a 2-step process: turn your data into valid XML, either using CDATA sections or by replacing < and & in the content of "elements" that contain HTML. BTW you also need to convert the charaters to UTF-8, or maybe to add an encoding declaration, my previous code works by pure accident, if you add a comma after the é the XML parser will complain (loudly!), then, and then only, you can use XML tools,
• a variant would be to use a custom parser that generates SAX events, then you can use XML SAX tools to process the data, and if you need to use real XML (or CSV, or any other format for which a SAX parser exists) you can just use the appropriate parser. Note that in order to generate SAX events you still need to escape & and < and probably to pass properly encoded strings to the SAX processor. Kip Hampton wrote an excellent column about this on xml.com: Writing SAX Drivers for Non-XML Data .

Another technique I've used (only for outputting XML with XML::Writer though) is to filter the data. In my case I tied the Handle XML::Writer was using. That tie filtered the data to make it XML compliant.

The power here (IMO) is that you're separating the filtering of HTML from the parsing of XML. You can blow up that HTML parsing independently as needed, again using existing tools like HTML::Parser. The problem with rolling your own is that it seems simple until you hit all the exceptions. Read all the Perl docs on why not to parse your own HTML as an example. Anything beyond the character-by-character filtering I describe above will fail miserably as the data changes to have tags spanning lines, nested tags, etc.

Filtering and using standard interfaces is an approach I prefer. It fits that UNIX-like philosophy of not reinventing the wheel and using existing tools as filters to coerce things into models that are predictable.

In short, leverage the work of others into the problem at hand by focusing only on the exceptions unique to your case.

For the same reasons, I was a bit scared off mirod's idea, there is anywhere between 10 and 30 columns I'm looking for so going through the data file and adding the CDATA sections seems a bit inefficient. I will bear this idea in mind, I'm tempted to try and change the data files to use these.

The best I can come up with is a single loop:

while (/<([^>]+)>/g) {
    if ($1 eq '/r') {  
        print Dumper(\%temp);
        undef %temp;  
    } else {
        next unless $wanted{$1};
        if (substr($1, 0, 1) eq '/') {
          my $text = substr($_,$pos,pos()-$pos-length($1)-2);     
           $temp{substr($1, 1)} = $text;
        } else {
            $pos = pos();
        }
    }

}
[download]

gav^

Otherwise, go to Why I like functional programming by tilly, and use his parser (the module in the thread) to build a custom one.

tillys code is a perfect example of why people shouldn't roll their own parser. It is a complete, and fairly complex one, and most people fail to cover all the bases.

Please do not try to roll your own if you wish to have a working solution very soon (unless you like doing things the hard way)

This has been a test of the emergency perl system, you must read pod now!

You can't use XML::Parser to parse everything, especially stuff that isn't really XML like my data. For example running this code -- Re: Is a file XML? proves it.

gav^