XML::Simple relies on other XML parsers. It can use XML::Parser or one of many SAX parsers. XML::Simple is at its fastest when it uses XML::Parser, so you should install XML::Parser and set

$XML::Simple::PREFERRED_PARSER = 'XML::Parser';
[download]

to make sure XML::Simple will use it. Hopefully, that will also bypass the problem you have with XML::SAX.