Segmentation fault from XML::Twig::Elt

isync has asked for the wisdom of the Perl Monks concerning the following question:

I am building XML sitemaps with XML::Twig::Elt. And I think am am handling it wrongly as it runs into a segmentation fault although it seems to be a proven module.

sub add_html {
        my $self = shift;
        my $ref = shift;

        # step over elements in fixed order
        my @elements;
        foreach my $key (qw(loc changefreq lastmod priority)){
                if($key eq 'loc'){
                        if($ref->{loc}){
                                push(@elements, XML::Twig::Elt->new('l
+oc', {}, $ref->{loc}) );
                        }else{
                                print "Error: 'loc' is a mandatory val
+ue and missing!\n"
                        }
                }elsif($key eq 'priority'){
                        if($ref->{priority}){
                                push(@elements, XML::Twig::Elt->new('p
+riority', {}, $ref->{priority}) );
                        }else{
                                print "Error: 'priority' is a mandator
+y value and missing!\n"
                        }
                }
        }

        # wrap these sub-elements into an "url"-level
        my $elt = XML::Twig::Elt->new('url', {}, @elements);
        undef(@elements);

        # and add this url-sub-element nest to the xml-document
        $elt->paste(last_child => $self->{xml});

        $self->{cnt}->{html}++;
}
[download]

As it seems I get the segfault on large structures, after a few thousand inserts, at least it segfaults after ~ 50000 calls to the code above. The function triggering the error is the ->paster() bit.

Could be XML::Twig::Elt has problems handling large structures. I would refactor my module then to write to file in smaller chunks. But before I do that I'd like to ask if someone sees what I am doing wrong here.

Any ideas?

Comment on Segmentation fault from XML::Twig::Elt Download Code

Replies are listed 'Best First'.
Re: Segmentation fault from XML::Twig::Elt by isync (Hermit) on Nov 07, 2011 at 00:08 UTC
The script ran through when I commented out the paste() line, that's how. And as you advised, I've updated my XML:: module stack, same error after that. So I pinned the script to the test bench, and here it is: segfaults after first sitemap: Read more... (7 kB)	[reply] [d/l]
Re^2: Segmentation fault from XML::Twig::Elt by Anonymous Monk on Nov 07, 2011 at 08:20 UTC
To clarify, I do not believe this to be an XML::Twig::Elt bug, after all, it is pure-perl. It is probably a perl bug, or c/xs bug in XML::Parser::Expat, Scalar::Util, XSLoader, Encode... In any case, this is not good :/ If you're on linux, you might give strace a shot, to try to pinpoint this further	[reply]
Re^2: Segmentation fault from XML::Twig::Elt by Anonymous Monk on Nov 07, 2011 at 08:13 UTC
I confirm a serious problem. I get premature end of program. No exceptions (segfault), program just ends for no reason before its finished. I has something to do with Scalar::Util::weaken I've whittled the proof down to #!/usr/bin/perl -- $Devel::Trace::TRACE = 0; # Disable use strict; use warnings; use XML::Twig; Main( @ARGV ); warn "main who\n"; exit( 0 ); sub Main { my $self = {}; $self->{xml} = XML::Twig::Elt->new('urlset', { 'xmlns' => 'http://www.google.com/schemas/sitemap/0.84' +, 'xmlns:xsi' => 'http://www.w3.org/2001/XMLSchema-instance', 'xsi:schemaLocation' => join(' ', 'http://www.google.com/schemas/sitemap/0.84', 'http://www.google.com/schemas/sitemap/0.84/sitemap.xsd', ), }); #~ <url> #~ <loc>http://www.example.com/some/really/interesting/file.html</ +loc> #~ <priority>0.5</priority> #~ </url> #~ for( 0 .. 149500 ){ #~ for( 0 .. 49500 ){ #~ for( 0 .. (49500-39500) ){ #~ for( 0 .. (10000 - 5000) ){ #~ for( 0 .. ( 5000 - 200 ) ){ #~ for( 0 .. ( 5000 - 140 ) ){ #~ for( 0 .. ( 5000 - 170 ) ){ #~ for( 0 .. ( 5000 - 180 ) ){ #~ for( 0 .. ( 5000 - 190 ) ){ #~ for( 0 .. ( 5000 - 183 ) ){ #~ for( 0 .. ( 5000 - 185 ) ){ #~ for( 0 .. ( 5000 - 183 ) ){ for( 0 .. 4817 ){ my @elements; push(@elements, XML::Twig::Elt->new('loc', {}, 'http://www.exa +mple.com/some/really/interesting/file.html' ) ); push(@elements, XML::Twig::Elt->new('priority', {}, '0.5' ) ); my $elt = XML::Twig::Elt->new('url', {}, @elements); $elt->paste(last_child => $self->{xml}); warn "main who $_\n" if $_ > ( 49497 - 3); } binmode STDOUT; #~ print $self->{xml}->sprint();# FAIL,incomplete $self->{xml}->set_pretty_print('indented'); my $header = '<?xml version="1.0" encoding="UTF-8"?>'."\n"; $self->{xml}->print_to_file('2'); # NO cutoff XML s, but still exi +ts on undef; #~ $self->{xml}->purge; #~ Can't call method "purge_up_to" on an undefined value at C:/perl/si +te/5.14.1/lib/XML/Twig.pm line 8087. #~ print $header; #~ $self->{xml}->print(); # XML is cut off #~ print $self->{xml}->sprint(); # XML is cut off, no closing #~ print $self->{xml}->flush; # fail #~ Can't call method "flush_up_to" on an undefined value at C:/perl/si +te/5.14.1/lib/XML/Twig.pm line 8082. warn "about to undef "; $Devel::Trace::TRACE = 1; # Enable undef $self->{xml}; warn "after undef "; return; } __END__ [download] The Devel::Trace shows sub XML::Twig::Elt::DESTROY checking whether to break circular reference manually, 4820 times (versus 4817 children), and skipping because we're using Scalar::Util::weaken `undef $self->{xml}; >> C:/perl/site/5.14.1/lib/XML/Twig.pm:7534: { my $elt= shift; >> C:/perl/site/5.14.1/lib/XML/Twig.pm:7535: return if( $XML::Tw +ig::weakrefs); >> C:/perl/site/5.14.1/lib/XML/Twig.pm:7534: { my $elt= shift; >> C:/perl/site/5.14.1/lib/XML/Twig.pm:7535: return if( $XML::Tw +ig::weakrefs); ...` [download] Adding `local $XML::Twig::weakrefs;` before undef doesn't change the problem. I'm using perl 5.14.1 on win32 $ perl -d:Modlist -e " use strict; use warnings; use XML::Twig; " Carp 1.23 Config DynaLoader 1.13 Encode 2.44 Encode::Alias 2.15 Encode::Config 2.05 Encode::Encoding 2.05 Exporter 5.65 Exporter::Heavy 5.65 File::Basename 2.82 File::Glob 1.12 File::Spec 3.33 File::Spec::Unix 3.33 File::Spec::Win32 3.33 List::Util 1.23 Scalar::Util 1.23 UNIVERSAL 1.08 XML::Parser 2.41 XML::Parser::Expat 2.41 XML::Twig 3.39 XSLoader 0.15 base 2.16 bytes 1.04 constant 1.21 feature 1.20 overload 1.13 utf8 1.09 vars 1.02 warnings 1.12 warnings::register 1.02 [download]	[reply] [d/l] [select]
Re^3: Segmentation fault from XML::Twig::Elt by mirod (Canon) on Nov 07, 2011 at 10:33 UTC
I can reproduce the problem in win32 (perl 5.12.3), but not on linux (perl5.12.3, 5.12.4 or 5.14.2). So since it's the same code on all 3 versions, I would think it's a bug somewhere in Scalar::Util::weaken or in the core, but only on windows. Which is a bit annoying since I don't develop on windows, I only use it to test modules before a release. I am not sure where to go from there honestly. Any idea?	[reply]
Re^4: Segmentation fault from XML::Twig::Elt by isync (Hermit) on Nov 07, 2011 at 14:07 UTC
Re^4: Segmentation fault from XML::Twig::Elt by Anonymous Monk on Nov 08, 2011 at 04:46 UTC
Re^5: Segmentation fault from XML::Twig::Elt by mirod (Canon) on Dec 06, 2011 at 18:36 UTC
Re: Segmentation fault from XML::Twig::Elt by mirod (Canon) on Nov 08, 2011 at 14:50 UTC
How much memory does your system have? I can reproduce the problem on linux by setting a small stack trace (2Mb fails, 4Mb doesn't). The amount of RAM is a difference between my linux and windows systems. The problem could be due to not having enough memory available after all.	[reply]
Re: Segmentation fault from XML::Twig::Elt by Anonymous Monk on Nov 06, 2011 at 22:22 UTC
The function triggering the error is the ->paster() bit. .... Could be XML::Twig::Elt has problems handling large structures. I would refactor my module then to write to file in smaller chunks. But before I do that I'd like to ask if someone sees what I am doing wrong here. How did you determine it is the paste bit that triggers the segfault? What happened when you upgraded all dependencies? You should write a program with the express purpose of trying to replicate this crash, say, ~50000 pastes of a similar sort	[reply]