in reply to xml parsers: do I need one?
[sam@localhost xml_test]$ time ./parser.pl > parser.out real 0m1.827s user 0m1.730s sys 0m0.040s [sam@localhost xml_test]$ time ./regex.pl > regex.out real 0m0.200s user 0m0.160s sys 0m0.040s [sam@localhost xml_test]$ diff parser.out regex.out
So I'm seeing a simple regex beating a simple XML::Parser implementation by around 9x. Given that your regex takes three minutes, an XML::Parser script taking 35 minutes is within a similar multiple. When you consider that XML::Parser is making multiple Perl sub calls on each element it encounters, I guess it makes sense. But, still, ouch!.
Of course, if it were my job on the line I'd still use an XML parser. I've been bitten by changing specifications and funky data too many times to take the easy way out in the parser. In fact, these days I parse my XML twice - first with Xerces/C++ for schema validation and second with XML::Simple for actual usage. Better safe than sorry!
-sam
For the record, here's my test setup. First, the data generator:
#!/usr/bin/perl -w print '<?xml version="1" encoding="UTF-8" ?>', "\n"; print "<test>\n"; for (0 .. 9000) { my $word = get_word(); print "<$word>\n"; if (rand(10) > 3) { for (0 .. rand(5)) { my $msg = get_words(30); print "\t<message>$msg</message>\n"; } } print "</$word>\n"; } print "</test>"; BEGIN { my @words; open(WORDS, "/usr/dict/words") or open(WORDS, "/usr/share/dict/words") or die "Can't open /usr/dict/words or /usr/share/dict/words: $ +!"; while (<WORDS>) { chomp; push @words, $_ if /^\w+$/; } srand (time ^ $$); # get a random word sub get_word { return lc $words[int(rand(scalar(@words)))]; } # get $num random words, joined by $sep, defaulting to " " sub get_words { my ($num, $sep) = @_; $sep = " " unless defined $sep and length $sep; return join($sep, map { get_word() } (0 .. ((int(rand($num)))+ +1))); } }
The regex parser:
#!/usr/bin/perl -w open(FILE, 'test.xml') or die $!; my $xml = join('', <FILE>); while($xml =~ m!<message>([\w\s]+)</message>!g) { print $1, "\n"; }
And the XML::Parser script:
#!/usr/bin/perl -w use strict; use XML::Parser; my $p = new XML::Parser(Style => 'Stream', Pkg => 'main'); $p->parsefile('test.xml'); my $in_msg = 1; sub StartTag { $in_msg++ if $_ eq '<message>'; } sub Text { print $_ if $in_msg; } sub EndTag { if ($_ eq '</message>') { $in_msg--; print "\n"; } }