hard to reply to this question: i have no idea if the issue is the algorithm or the memory which swap. I'll go a bit OT and try to suggest you to change completely your approach:

use a SAX parser and collect data while reading the XML only once. This could probably speed-up the process and reduce the mem usage and is a good hint if you are managing huge xml.

HTH