jasonv has asked for the wisdom of the Perl Monks concerning the following question:

I have a set of XML files to process, and currently I am using a DOM-oriented method to do so. However, because the files could get fairly large I'd like another method of processing them that doesn't mean putting the entire structure into memory.
A colleague suggested I use SAX and in particular SAX::Machines, which looks like it would do what I want to do quite easily. I have been looking for some examples of how to use SAX::Machines as well as the docs, and havent been coming up with much. My attempts to solve the problem using XML::Filter::DocSplitter and the other Machines modules have come to naught also.
My problem is that I am having real trouble getting into the SAX "frame of mind". I know what the problem is, I know a basic plan of action, but translating that into SAX code has turned out to be a real problem.
Here's a representitive of the structure of my XML: 
<root>
  <name>Parent Name</name>
  <node><name>Node 1</name>
  <node><name>Node 2</name>
    <child id="2263583">
      <start>
        <thing value="20001231945"/>
      </start>
    </child>
  </node></node>
  <node><name>Node 3</name>
  <node><name>Node 4</name>
    <child id="2274693">
      <start>
        <thing value="20001231930"/>
      </start>
    </child>
  </node>
 </node>
</root>

[download]
What I want to do is 'cut' the document into chunks at the deepest elements called 'node' (there can be either 1, 2 or 3 levels), take the contents of it (1 or more elements called 'child', and output those contents to a file name that is determined by what the values of the 'name' elements in it's ancestors are.
For example, the first 'child' element would be put into a file called 'ParentName_Node1_Node2.xml' or similar. So, the 'child' elements have to be aware of where they are in the hierarchy so they can be put into the relevant file. The last point that complicates it a little is that the xml should be placed as the last element of a file of the name IF it already exists; so that particular point stops me from using an 'in-memory' model, and suggests one that is stream or pipeline oriented (to avoid putting that file in memory).
So... I have been battling with this for a while now, and I see that my frame of mind seems to be wrong to solve the problem. Can somebody please point me in the right direction ?

Replies are listed 'Best First'.
Re: Getting into a SAX frame of mind
by mirod (Canon) on Jan 23, 2004 at 17:07 UTC

    XML::Twig is especially designed to work with big files. What you want to do can be done with the following code:

    #!/usr/bin/perl -w
use strict;
use XML::Twig;

my $t= XML::Twig->new( twig_handlers => 
        { node => sub { my( $t, $node)= @_;
                        return unless( $node->has_children( 'child'));
                        my $name= $node->first_child( 'name');
                        my $file= $name->text;
                        $name->delete;
                        print "in file $file\n"; # in real code open f
+ile
                        $node->print;            # in real code print 
+to file
                        print "\n\n";
                        $t->purge;               # get rid of the node
                      }
       })->parse( \*DATA);
                          

__DATA__
<root>
  <name>Parent Name</name>
  <node><name>Node 1</name>
  <node><name>Node 2</name>
    <child id="2263583">
      <start>
        <thing value="20001231945"/>
      </start>
    </child>
  </node></node>
  <node><name>Node 3</name>
  <node><name>Node 4</name>
    <child id="2274693">
      <start>
        <thing value="20001231930"/>
      </start>
    </child>
  </node>
 </node>
</root>

    [download]
[reply]
[d/l]

      You missed the requirement of having the file name derived from the names of ancestors. You'd need to add someting like:

      $file = join '_', map { $_->first_child_text('name') } $node->ancestor
+s( 'node' );    # still need to add name from root/name

      [download]

      (Which I know you know, but for the OP's benefit.)

      qq, a Twig fan

[reply]
[d/l]

        Oops! I missed the 's' at the end of 'elements' Thanks.

        I forgot to mention BTW that the nice thing about the above code is that at most you keep one node in memory.

[reply]

Back to Seekers of Perl Wisdom