In case you want to stick with XML::Simple (to avoid having to change existing code), you might want to check which parser XML::Simple is using under the hood  (in case of doubt, print out which modules have been loaded at the end of your script, using print join "\n", sort values %INC;)

As you're already using XML::LibXML anyway, you might want to try telling XML::Simple to use it by setting:

$XML::Simple::PREFERRED_PARSER = "XML::LibXML::SAX";
[download]

(See XML::Simple Environment for the details.)

As XML::LibXML is known to be one of the fastest XML parsers, this might speed up things (even though XML::Simple would of course still be creating its gazillion of hashes and arrays...)