Hello Monks,

I am looking for some guidance on this project below, which will involve parsing a directory of XML files one by one, checking for and eliminating duplicate "article" results.

I've managed to get perl reading through a directory, grabbing and parsing files, and printing out data, but now I have input buffer memory problems with the function for eliminating duplicate elements. I'm guessing a variable is getting out of control, but I'm flushing out the twig, so I am not sure where I've gone wrong. The program fails with the following message: "Ran out of memory for input buffer at /usr/lib/perl5/XML/Parser/Expat.pm line 469. at xml_result.pl line 35 at xml_result.pl line 35".

I put a tar archive of a sample of the data here. And here is the code.

Regards.

Leigh

#!/usr/bin/perl

# turn on perl safety features
        use strict;
        use warnings;

#initialize modules
        use XML::Twig;
        use Data::Dumper;
        use DirHandle;
        use Digest::MD5;

# check for working directory

my $dir = $ARGV[0] or die "Must specify directory on command line";
my($filepath) = { };                    # declare file list
my %md5;                                # file to elim duplicates




my @filepath_list = xmlfiles($dir);     # call xmlfiles subroutine
                                        # to get list of files from da
+ta dir
print "Processed files: \n";            # print list of processed
foreach (@filepath_list) {              # files
  print "$_\n";
}

foreach $filepath (@filepath_list) {

        (my $outfile = $filepath)=~ s{\.xml$}{.clean.xml}; # dest. fil
+e
        open( OUT, ">$outfile") or die "cannot create output file!";
        my $twig = new XML::Twig(
                twig_handlers => { article => \&eliminate_dup });
        $twig->parsefile($filepath);
        $twig->flush(\*OUT);                            #save memory
        close OUT;                                    #close file
}

exit;

sub xmlfiles
{
        $dir = shift;
        print $dir, "\n";
        my $dh = DirHandle->new($dir) or die "can't open directory";
        return sort                      # sort pathnames
                grep {    -f     }       # choose only files
                map  { "$dir/$_" }       # create full paths
                grep {  !/^\./   }       # filter out dot files
                $dh->read();             # read all filenames
}

sub eliminate_dup
{
        my( $t, $elt)= @_;
        my $elt_text= $elt->sprint;     # get text and tags
        my $md5= md5($elt_text);
        if( $md5{$md5}) {
                $elt->delete; }         # if md5 exists, remove elemen
+t
        else {
                $md5{$md5}=1; # store md5
                $t->flush( \*OUT);      # flush memory
        }
}

[download]

In reply to Memory problems parsing XML by leighgable

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.