Debugging perl crash

vsespb has asked for the wisdom of the Perl Monks concerning the following question:

I have errors (during global destruction - exit()) like this:

Attempt to free nonexistent shared string '', Perl interpreter: 0x801c
+19c00 at ./gemu-test-worker.pl line 1311, <STDIN> line 1.
Not a CODE reference at ./gemu-test-worker.pl line 1311, <STDIN> line 
+1.
END failed--call queue aborted at ./gemu-test-worker.pl line 1311, <ST
+DIN> line 1.
Segmentation fault (core dumped)
[download]

panic: del_backref, *svp=801e59fc0, sv=8048dd5d0 at /ara/backup/disast
+er_backup/glacier/mt-aws/lib/App/MtAws.pm line 120.

Not a CODE reference at /ara/backup/disaster_backup/glacier/mt-aws/lib
+/App/MtAws.pm line 122.
END failed--call queue aborted at ./glacier/mt-aws/mtglacier line 122.
panic: invalid MRO! at ./glacier/mt-aws/mtglacier line 122 during glob
+al destruction.
[download]

sometimes on one machine there is stacktrace


Core dump stack:

(gdb) backtrace
#0 0x00000008008731f5 in Perl_Gv_AMupdate ()
from /usr/local/lib/perl5/5.16/mach/CORE/libperl.so
#1 0x0000000800873508 in Perl_gv_handler ()
from /usr/local/lib/perl5/5.16/mach/CORE/libperl.so
#2 0x00000008008e6514 in S_curse ()
from /usr/local/lib/perl5/5.16/mach/CORE/libperl.so
#3 0x00000008008e5b92 in Perl_sv_clear ()
from /usr/local/lib/perl5/5.16/mach/CORE/libperl.so
#4 0x00000008008dc369 in Perl_sv_clean_objs ()
from /usr/local/lib/perl5/5.16/mach/CORE/libperl.so
#5 0x00000008008669c7 in perl_destruct ()
from /usr/local/lib/perl5/5.16/mach/CORE/libperl.so
#6 0x0000000000400f39 in main ()
[download]

or simply Not a CODE reference + END failed--call queue aborted
or weird things like in "print Dumper \%INC" output line

          'base.pm' => '/usr/local/lib/perl5/5.16/base.pm',
[download]

replaced with

'' => '',
[download]

(and message "Attempt to free nonexistent shared string" thrown)
or things just hangs with 100% CPU (and Ctrl-C produces Segfault)
happends in two different independent scripts, reproduced by me and another person, under FreeBSD 10.x only (perl 5.16.3). on two different machines.
Not reproducible under Linux in same perl version. In both cases error 100% reproducible.
In both cases scripts work (read,write) large files (> 4Gb) by chunks (i.e. not whole file at once).
Attempt to reduce file size throw error away. Attempt to reduce script to proof-of-concept is failed too (problem that script run is too long, so had to wait 10+mins after each modification to see if it still reproducible)

Note that scripts are not using threads. And both scripts fork other processes.
Question is - how to effectively debug such kind of things? What this can be? What to check?

UPD: It seems FreeBSD uses clang by default: http://unix.stackexchange.com/q/49906

$ cc --version
FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
Target: x86_64-unknown-freebsd10.0
Thread model: posix
[download]

so this is perl compilled by clang

UPD2: finaly created a PoC and reported a bug RT#122199 seems not only clang affected. reproduced with 5.18 gcc (but 5.16 clang only)

Comment on Debugging perl crash Select or Download Code

Replies are listed 'Best First'.
Re: Debugging perl crash by dave_the_m (Monsignor) on Jun 27, 2014 at 15:32 UTC
The stuff you're seeing is most likely due to something earlier corrupting the state of the perl interpreter, which is going to be hard to track down. This will be either due to a bug in the perl core itself, or due to a bug in an XS-based module you are loading. If the latter, see if there's a newer release of that module; if the former, try using perl 5.20.0; in either case, the bug may have been found and fixed. Dave.	[reply]
Re^2: Debugging perl crash by vsespb (Chaplain) on Jun 27, 2014 at 15:49 UTC
Thanks. I prefer to try to find exact reason (and PoC code) for this bug and either find workaround or report it to FreeBSD team. Otherwise my software will be unusable (with stock perl) under whole family of FreeBSD 10+ and maybe other systems using llvm+perl 5.16(+?)	[reply]
Re^3: Debugging perl crash by dave_the_m (Monsignor) on Jun 27, 2014 at 17:43 UTC
I prefer to try to find exact reason In that case you're going to need to become an expert in the internals of perl and do a lot of C level debugging. I'd strongly recommend before you go that route, of trying to reduce your code to a managable size, and in particular, to eliminate as many external dependencies as possible. At that point you will either have eliminated all XS-based modules (so the bug is in perl), or will have found out which XS module has the bug. Trying with with newer versions of perl or the module will show whether the bug has been fixed (in which case you can bisect to see what patch fixed the issue), or if the bug is still present, will show you who to notify. Dave.	[reply]
Re: Debugging perl crash by 1s44c (Scribe) on Jun 28, 2014 at 23:55 UTC
This may be a compiler issue. Clang/LLVM is the future but it's not been as heavily tested with perl as GCC has. Just maybe something is off. Try building perl with gcc and see if the problem goes away. `cd /usr/ports/lang/gcc make install clean export CC=gcc .. Rebuild perl. Sounds like you are not using a port version..` [download] I do have some heavy stuff running under perl v5.14.4 compiled by default clang on FreeBSD 10/x86_64. I've never seen crashes like you describe.	[reply] [d/l]
Re^2: Debugging perl crash by vsespb (Chaplain) on Jun 29, 2014 at 16:53 UTC
Yes, cannot reproduce under gcc. Clang only.	[reply]
Re^3: Debugging perl crash by 1s44c (Scribe) on Jun 30, 2014 at 11:03 UTC
Compile it with reduced optimization under clang and see if that makes the problem go away? At least that's a starting point to track down the real problem. Can you come up with a simple perl program that shows the issue? EDIT: Maybe this is more a thing for bugs.perl.org than perlmonks? It's certainly an interpreter and/or compiler bug.	[reply]
Re^4: Debugging perl crash by vsespb (Chaplain) on Jun 30, 2014 at 11:42 UTC
Re^4: Debugging perl crash by vsespb (Chaplain) on Jun 30, 2014 at 17:36 UTC
Re: Debugging perl crash by perlfan (Parson) on Jul 02, 2014 at 12:35 UTC
You may want to visit #bsdperl on irc.perl.org as well, so says https://wiki.freebsd.org/Perl#IRC FWIW, I remain unhappy with FreeBSD moving to clang. But I am not 100% this is a clang issue, it may very well be a FreeBSD 10.x issue. Have you tried it on 9.x or 8.x?	[reply]
Re^2: Debugging perl crash by vsespb (Chaplain) on Jul 02, 2014 at 14:53 UTC
Tried FreeBSD 9 perl 5.12.4 gcc, cannot reproduce (well, the fact that I cannot reproduce does not mean it's not exists, it's sometimes hard to reproduce in on FreeBSD10 after reboot)	[reply]