inputsprocket has asked for the wisdom of the Perl Monks concerning the following question:

Hi folks,
I've been using XML::Simple for all my XML parsing needs in the past, but I need to start parsing complicated XML files >50MB in size, and well, I run out of system memory before I get anywhere.

So I've been looking into Twig, but I ran out of brain memory before I got anywhere.

Can someone please explain how I extract all <node> elements in this XML eg knowing that <node id="xx"> can have any number of <node> children, recursively.
I basically need to get to the first <node>, extract all the data from the elements and when I get to a child <node> do the same. Rinse&Repeat


eg, a snippet from my xml: 
<data>
<geography> 
    <node id="0" hidden="0">
        <description>By Geography</description> 
        <ticketsavailable>1</ticketsavailable> 
        <parent_id>-1</parent_id> 
        <parent_path>,0</parent_path> 

        <node id="709" hidden="0"> 
            <description>Alabama</description> 
            <title>Tickets for Events at Alabama Venues</title> 
            <meta_description>Tickets for Events at Alabama Venues</me
+ta_description> 
            <seo_title>Tickets for Events at Alabama Venues</seo_title
+> 
            <seo_description>Tickets for Events at Alabama Venues</seo
+_description> 
            <ticketsavailable>1</ticketsavailable> 
            <parent_id>0</parent_id> 
            <parent_path>,0,709</parent_path> 

            <node id="4945" hidden="0"> 
                <description>Birmingham</description> 
                <title>Birmingham Tickets</title> 
                <meta_description>Birmingham Tickets</meta_description
+> 
                <keywords>Birmingham tickets, Alabama tickets, Birming
+ham events, tickets for events in Birmingham and Alabama</keywords>
                <seo_title>Birmingham Tickets</seo_title>
                <seo_description>Birmingham Tickets</seo_description>
                <name_primary>Birmingham</name_primary>
                <ticketsavailable>1</ticketsavailable>
                <parent_id>709</parent_id> 
                <parent_path>,0,709,4945</parent_path> 
                
                <node id="7983" hidden="0"> 
                    <description>Alabama Theatre</description> 
                    <title>Alabama Theatre Tickets at StubHub!</title>
+ 
                    <meta_description>Alabama Theatre Tickets - Buy an
+d sell tickets to events at Alabama Theatre in Birmingham, AL at Stub
+Hub!</meta_description> 
                    <keywords>Alabama Theatre seating chart,AlabamaThe
+atre, Alabama Theatre tickets</keywords> 
                    <seo_title>Alabama Theatre Tickets</seo_title> 
                    <name_primary>Alabama Theatre</name_primary> 
                    <ticketsavailable>1</ticketsavailable> 
                    <parent_id>4945</parent_id> 
                    <parent_path>,0,709,4945,7983</parent_path> 
                    <venue_addr1>1811 3rd Ave. North</venue_addr1> 
                    <venue_city>Birmingham</venue_city> 
                    <venue_state>AL</venue_state> 
                    <venue_zip>35201</venue_zip> 
                    <venue_phone>2052522862</venue_phone> 
                    <venue_genre genre_id="1632"> 
                        <venue_genre_event event_id="337496"/> 
                    </venue_genre> <venue_genre genre_id="75215"> 
                        <venue_genre_event event_id="331248"/> 
                    </venue_genre> 
                    <venue_genre genre_id="99217"> 
                        <venue_genre_event event_id="330232"/> 
                    </venue_genre> 
                    <venue_genre genre_id="130035"> 
                        <venue_genre_event event_id="342530"/> 
                    </venue_genre> 
                </node> 
                    
                ...etc
                
            </node>
        </node>    
    </node>
</geography>
</data>

[download]

As I don't know how many <node> elements are nested, I am completely lost.

Any help very much appreciated!

Edited by planetscape - added readmore tags

( keep:5 edit:17 reap:0 )

Replies are listed 'Best First'.
Re: Recursive XML navigation XML::Twig help!
by GrandFather (Saint) on Sep 30, 2006 at 20:45 UTC
[reply]
Re: Recursive XML navigation XML::Twig help!
by inputsprocket (Initiate) on Oct 01, 2006 at 09:13 UTC
    Just to add,
    I can recursively extract all the elements, but the problem <node id="0"> comes last, ie Twig is performing the entire parsing in memory (<geography> node is 35MB large) and runs out of memory...

    What I want, is to parse <node id="0"> and purge before moving onto the next node

    what I have at the moment, which is running out of memory is

    
#! /usr/bin/perl

use strict;
use warnings;
use XML::Twig;

my $file = $ARGV[0];

my $twig= new XML::Twig(
    TwigRoots=>{geography=>1},
    TwigHandlers=>    {    'node' =>\&geog_node
                    }
    
);

$twig->parsefile($file);

sub geog_node {
    my($twig, $section) = @_;
    if($section->att('hidden') eq '0') {
        print"Data for node: \n";
        print"\tID:\t",$section->att('id'),"\n";
        print "\tDesc:\t",$section->first_child('description')->text,"
+\n";
    }
}

    [download]
[reply]
[d/l]

      With XML::Twig, usually you free the memory by purge-ing the twig after you're done with the data. Of course your problem here is that after you're done with the data in the inner node you can't use purge, as it would remove data about the outter node, that you still need to use.

      I see 2 alternatives: if you just delete a node after you're done displaying it, then you're good: by definition all of its inner nodes have been processed (and indeed deleted), so you can delete it, memory is freed, everybody's happy. Just add $section->delete at the end of geog_node.

      The other alternative, is a little more complex, but has the advantage of outputting the nodes in the order in which they appear in the document (starting wit the first outter node): when you get to the first node to be processed, which will be an inner node, you already have all the information available to output all of its ancestors (their id and description have already been parsed)... so just do it. Go back to the ancestors and output them one by one. Mark them as output so you don't do it again for the next inner node... et voilà! At that point you can safely purge the twig, you've used all of its relevant info.

      #! /usr/bin/perl

use strict;
use warnings;
use XML::Twig;

my $file = $ARGV[0];

my $twig= new XML::Twig(
    TwigRoots=>{geography=>1},
    TwigHandlers=>    {    'node' =>\&geog_node
                    }
    
);

$twig->parsefile($file);

sub geog_node {
    my($twig, $node) = @_;
    # ancestors_or_self returns the node first, the ancestors from inn
+er to outter
    # hence the reverse
    foreach my $node_to_output ( reverse $node->ancestors_or_self( 'no
+de[!@output]')) { 
        if($node_to_output->att('hidden') eq '0') { 
            print node_summary( $node_to_output); 
        }
        $node_to_output->set_att( output => 1); # mark so we don't try
+ to output them again
    }
    $twig->purge;                               # now it's safe to pur
+ge 
}

sub node_summary
  { my $node= shift;
    return sprintf "Data for node:\tID:\t%s\n\tDesc:\t%s\n", $node->at
+t('id'),
                                                             $node->fi
+eld('description')
                                                             ;
  }

      [download]
[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom