I would try converting the SGML part of the message in XML so you can use XML tools on it.
Converting SGML to XML can be very easy... or a real PITA, depending on the usage of SGML-specific features used in the SGML. And I am not even talking obscure stuff, but just things like & foo which is valid SGML (the space after the entity allows an SGML parser to infer the missing ';').
So here are your options:
- the easy and dangerous one: use regexps to add the missing tags at the end of lines: easy because you can pobably hack something that works in a couple of minutes, dangerous because there is 0 garantee that the output is indeed XML, and if the bank starts using different (legal) features of SGML (ah the guilty pleasure of using a CONREF!) you'll be stuck,
- The easy and right way: use James Clarke's sx which oddly enough does SGML to XML conversion, and considering James'nsgml is the reference parser for SGML and that he wrote half of the XML spec I would think it does a good job,
- the hard but right (and hubristic!) way: write a SAX parser for SGML, using a wrapper around the SP library, so you can then use any XML tool that supports a SAX input, such as the last version of XML::Simple.