in reply to XPath crashing

Loading 32MB of XML into a DOM structure is bound to use quite a lot of memory, but it seems the XML::XPath->find() is buggy. In either case you'd be better of using something that doesn't build the whole structure, expecially if as you said in the other node you only need very little out of the large XML. XML::Twig or maybe XML::Rules ;-)

It's hard to give you an example without knowing the XML, but using XML::Rules you might start with: 

my $parser = XML::Rules->new(
 rules => [
  'computers,and,other,tags,you do not care about at all' => 'skip',
  '^user' => sub {return $_[1]->{teamid} eq $CONFIG{teamid} },
  'user' => 'as array',
   # only the <user> tags containing the right attribute will be proce
+ssed
  '_default' => 'as is',
  'tags,with,no,attributes' => 'content',
 ]
);

my $data = $parser->parse($the_xml);

[download]
And you end up with a data structure containing only the content of the <user> tags you are interested in. And at no time is the whole document in memory, the module only keeps the data you wanted and the attributes of the yet unclosed nodes in memory.

Jenda
Support Denmark!
Defend the free world!

Replies are listed 'Best First'.
Re^2: XPath crashing
by adrianxw (Acolyte) on Feb 05, 2007 at 20:47 UTC
    Thanks for the steers, I have been browsing the Twig and Rules documentation today. I will download the packs this evening and experiment with them.

    The xml is very simple... 

    <users>
 <user>
  5-6 simple key's here
  <teamid>100</teamid>
 </user>
</users>

    [download]
    ... there are many thousands of user tags, but I am only interested in the 100 - 150 records with the correct teamid.
[reply]
[d/l]

      It's a shame the teamid is a subtag and not an attribute, you will be unable to use the "start" rules. Which will make the script a bit slower, but not more memory hungry. In this case I think 

      my $parser = XML::Rules->new(
 rules => [
  _default => 'content',
  user => sub {
   return unless $_[1]->{teamid} = $_[4]->{parameters}{teamid};

   delete $_[1]->{_content}; # delete the textual context (whitespace)
   return '@user' => $_[1];
  },
  users => 'pass no content',
 ]
);

my $users = $parser->parsefile( $the_xml_file, {teamid => 100};

      [download]
      should give you the data for the right users only.

      Jenda
      Support Denmark!
      Defend the free world!

[reply]
[d/l]

In Section Seekers of Perl Wisdom