I have been down that road a couple of times, and ended up with ugly regexps that tried to identify whether < or & where part of the markup or needed to be escaped. Basically a < that's not followed by a letter should probably be escaped, and a & that's not followed by (#x?\d+|\w+;) should be escaped. Be sure to trace what you replace so you can spot problems.

Down this path lies madness though. If the provider of the data claims it's XML, then you usually have a good leverage to force them to fix it at the source. That's the sanest way to go. A little work on their part (maybe you can help them) will save you and eventually them lots of headaches down the road.

Just for the fun, I have actually used an other (wrong) option: provided the XML is close enough to SGML, and has a DTD (or you can write its DTD easily), you can try using sx (also called osx in some linux distributions) to convert the SGML to XML. SGML is actually much more lax about what needs to be escaped, the parser will try to figure out whether a < or & is a separate token, or part of the markup. But once again that's just a stop gap (and probably quite a hard one to set-up), try to get the "quasi-XML" to be XML, and spend your time doing useful things instead of fixing other people's mistakes.

But of course any decent XML-parser (such as XML::Parser) will choke on this "bad" XML (I wonder if technically it is even XML due to the missing encoding of 'forbidden' characters).

Therefore I suggest that you try to capture these 'forbidden' characters before they enter your XML. Can't you run the encode-function of HTML::Entities on the incoming data, prior to it being XML-ized?

#!/usr/bin/perl -w
use strict;
my $incoming_xml = '
<?xml version="1.0" encoding="UTF-8"?>
<TRANSACTION>
    <FIELDS>
        <FIELD KEY="user">name</FIELD>
        <FIELD KEY="password">pass&word</FIELD>
        <FIELD KEY="operation_type"><do_what></FIELD>
     </FIELDS>
</TRANSACTION>';
print $incoming_xml;
print "\n==============\n";
$incoming_xml =~ s/(<FIELD[^>]*>)(.*)(<\/FIELD>)/$1.&html_transliterat
+e($2).$3/eg;
print $incoming_xml;
exit;
sub html_transliterate{
    my $in_str = shift;
    $in_str =~ s/&/!38/g;
    $in_str =~ s/</!40/g;
    $in_str =~ s/>/!41/g;
    return $in_str;
}
[download]

<?xml version="1.0" encoding="UTF-8"?>
<TRANSACTION>
    <FIELDS>
        <FIELD KEY="user">name</FIELD>
        <FIELD KEY="password">pass&word</FIELD>
        <FIELD KEY="operation_type"><do_what></FIELD>
     </FIELDS>
</TRANSACTION>
==============

<?xml version="1.0" encoding="UTF-8"?>
<TRANSACTION>
    <FIELDS>
        <FIELD KEY="user">name</FIELD>
        <FIELD KEY="password">pass!38word</FIELD>
        <FIELD KEY="operation_type">!40do_what!41</FIELD>
     </FIELDS>
</TRANSACTION>
[download]

thanks that is exactly what i was looking for. as usual perlmonks come through again.

Valid XML only needs & and < to be turned into &amp; and &lt; respectively. You don't have to encode ' " or > at all.

Replacing all & by &amp; is trivial. This still leaves you with the troublesome <, but I don't think there is any possibility (other than validating your XML against a DTD) to let the script differentiate between <FIELDS> and <do_what>: they both look like a "valid" XML tag.

Oh yes, just remembered: ]]> must be encoded as ]]&gt; if you want to use it as text.

That's not true. You do have to encode ' and " if you expect to use them inside a quoted string using that character as a delimiter. So <foo bar=" &quot; "/> and <foo bar=' &apos; '/>

Perhaps, but it has nothing to do with it being valid XML or not: it is just about having balanced quotes.

CountZero

"If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law