Hello, I'm really hoping someone here can help me out with this...essentially I'm writing a script to search through a huge (400 MB) XML document for nodes where a certain child element's text matches some string. I have tried approaching this by loading each full node into memory, checking the appropriate child node's text, and either printing it or purging it based on a match or not. However, this ends up consuming a huge amount of memory, as there are over 400,000 records, and the purge method still keeps the reference of the root->main_element in memory. Also, it takes forever, and my computer slows down drastically.

Here's a condensed version of that approach...(FullSearch.pl)

==============================================

use strict;
use XML::Twig;

my $file = "bin/Example.xml";
my $node = $ARGV[0];
my $value = $ARGV[1];
my $i = 0;

my $twig= new XML::Twig( twig_handlers => { Record => \&Record });

$twig->parsefile($file);
print "Finished Search!\n";

sub Record { 
my( $twig, $record)= @_;
    my @matcharray= $record->get_xpath($node);
    for my $match (@matcharray[-1]){
    if( $match->text eq $value){
# PRINT ALL NODES FROM THIS RECORD HERE...
}}
$twig->purge;
}

[download]
==============================================

Should that not take up a ton of system resources?

So I thought, what if I used a start_tag_handler to keep track of which main_element record I'm on, then use twig_roots to only compare the search value against the specific child node I'm interested in. If there is a match, store the record number, otherwise, purge. This seems to work a whole lot faster, and consumes nominal memory.

Here's that approach...(MarkMatches.pl)

==============================================

use strict;
use XML::Twig;

my $file = "bin/Example.xml";
my $node = $ARGV[0];
my $value = $ARGV[1];
my $count = 0;

open(INDEX, ">bin/Index.txt");

my $twig = XML::Twig->new( 
start_tag_handlers => { Record => \&Count },
twig_roots => { $node => \&Record } );

$twig->parsefile($file);
$twig->purge;

sub Count {
my ( $twig, $element )=@_;
$count++;
$twig->purge;}

sub Record {
my ( $twig, $record )=@_;
if($record->text eq $value) {
    print INDEX $count."\n"; }
$twig->purge; }

[download]
==============================================

Right now, I am saving a list of matching records to a separate text file, then pulling that file into an array in a second script. This script does a start_tag_handler method again to get the current record count, and either lets it get processed by the twig_handlers to print the record if the count is in the array, or ignores it altogether (using no memory).

Here's that script...(ReturnRecords.pl)

==============================================

use strict;
use XML::Twig;

my $file = "bin/Example.xml";
my $i;
my $count = 0;

my $data_file = "bin/Index.txt";
open(DAT, $data_file) || die("Could not open file!");
my @id=<DAT>; 
close(DAT);

my $twig = XML::Twig->new(
start_tag_handlers => { Record => \&Check },
twig_handlers => { Record => \&Process }
);

$twig->parsefile($file);
$twig->purge;

sub Check { 
my ($twig, $record)=@_;
$i = 0;
$count++;
if ($count > $id[-1]){
    $twig->finish_now;}
my $thisId = $count."\n";
foreach my $id (@id){
if ($thisId eq $id){ 
    $i = 1; 
    last;}}
if ( $i ne 1){ 
    $record->ignore; }
}

sub Process { 
my ($twig, $record)=@_;
# PRINT ALL NODES FROM RECORD HERE...
$twig->purge;
}

[download]
==============================================

Is there a way I can combine these two scripts somehow? Could I maybe start rescanning the document within the first script? If I get a match with twig_roots, the entire twig consists of just the direct path to that one node. Is there a way to fully parse nodes where twig_roots matches? I can go backwards from the matching twig_root to the parent, but the only descendants available are the ones that made up the twig to begin with.

Also, all of the main record elements have an ID associated with them (see below). Is there a way to jump to a matching ID without scanning each one, even without using start_tag_handlers and comparing the ID?

Here's how the XML file is set up...(Example.xml)

==============================================

<Root>
   <Record id="1">
      <Title>Title 1<\Title>
      <Year>2007<\Year>
      <Author>W. T. Wright<\Author>
      <Copyright>
         <Year>2006<\Year>
         <Number>A84LEU<\Number>
      <\Copyright>
      …
      <Info>Blah Blah Blah<\Info>
   <\Record>
   …
   <Record id="429000">
      <Title>Title 999<\Title>
      <Year>2004<\Year>
      <Author>A. R. Smith<\Author>
      <Copyright>
         <Year>2003<\Year>
         <Number>D93YAK<\Number>
      <\Copyright>
      …
      <Info>Halb Halb Halb<\Info>
   <\Record>
<\Root>

[download]
Sorry this is so long. I've been working on optimizing this for a while, and just can't seem to get it to be as efficient as I need it to be. Right now it takes about 10-15 minutes to scan the document each time, or 20-30 minutes doing the "full search" (but the PC freezes, so that's not even an option).

Thanks!

In reply to XML::Twig questions by r1_fiend

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.