...Is there a way to pass only part of XML files to parse using this module

I think so, yes:
---

SYNOPSIS
           use XML::Simple;

           my $ref = XMLin([<xml file or string>] [, <options>]);
[download]

So if you have a way to cut out only parts of the original XML file, you can feed XML::Simple with that. Hard to say however if that would really speed up things.

Would it be possible to make available the problematic XML or something similar for testing ? It might allow some benchmarking and/or insight.