Hello Monks, I have a (multi-threaded) perl script that is occassionally seg faulting. I seek your wisdom in nailing the offending part of the script. 
Environment:
         OS: CentOS 6.4
       Perl: v5.10.1 (*) built for x86_64-linux-thread-multi
        DBI: 1.623
 DBD::mysql: 4.022
    Threads: 1.72

[download]

Script overview: This is the master script that is terminating with a seg fault. It spawns a number of threads each invoking an external shell script using system() call (and these shell scripts in-turn invoke external programs).It also creates an extra detached thread that periodically connects to a remote database to monitor the status of a few external programs (that report to the DB).

When this script terminates, I see all the shells scripts and the programs they spawned (all all their offsprings) running ok. Its only this master script and the monitor thread that it creates that don't exist anymore (expected after a seg fault).

The issue gets complicated because this failure is not consistent across runs. It only happens occasionally. Here is a gdb backtrace of the core dump generated by the failure (I've changed the credential info for obvious reasons, but the trace is left as is).

Any pointers in the right direction? 

Core was generated by `perl -d problem_script.pl 970 10.30.30.42 50,51
+,52,53,54,55,56,57,58,59'.
Program terminated with signal 6, Aborted.
#0  0x00000032fe6306f7 in kill () from /lib64/libc.so.6
(gdb) bt
#0  0x00000032fe6306f7 in kill () from /lib64/libc.so.6
#1  0x00000000004d8367 in Perl_apply ()
#2  0x00000000004c99c9 in Perl_pp_chown ()
#3  0x000000000048765e in Perl_runops_standard ()
#4  0x0000000000430cb4 in Perl_call_sv ()
#5  0x0000000000478166 in Perl_sighandler ()
#6  <signal handler called>
#7  my_stat (path=0x7fff24f5db80 "/usr/share/mysql/charsets/Index.xml"
+, stat_area=0x7fff24f5da50, my_flags=0)
    at /export/home/pb2/build/sb_0-10180689-1378750483.08/rpm/BUILD/my
+sql-5.5.34/mysql-5.5.34/mysys/my_lib.c:415
#8  0x00002ae284d8cd74 in my_read_charset_file (filename=0x2 <Address 
+0x2 out of bounds>, myflags=0)
    at /export/home/pb2/build/sb_0-10180689-1378750483.08/rpm/BUILD/my
+sql-5.5.34/mysql-5.5.34/mysys/charset.c:364
#9  0x00002ae284d8d615 in init_available_charsets () at /export/home/p
+b2/build/sb_0-10180689-1378750483.08/rpm/BUILD/mysql-5.5.34/mysql-5.5
+.34/mysys/charset.c:458
#10 0x00000032ff60c1d3 in pthread_once () from /lib64/libpthread.so.0
#11 0x00002ae284d8de48 in get_charset_by_csname (cs_name=0x199e2bf0 "l
+atin1", cs_flags=32, flags=2034312912)
    at /export/home/pb2/build/sb_0-10180689-1378750483.08/rpm/BUILD/my
+sql-5.5.34/mysql-5.5.34/mysys/charset.c:644
#12 0x00002ae284d7383f in mysql_set_character_set_with_default_collati
+on (mysql=0x19a48f00)
    at /export/home/pb2/build/sb_0-10180689-1378750483.08/rpm/BUILD/my
+sql-5.5.34/mysql-5.5.34/sql-common/client.c:2233
#13 mysql_init_character_set (mysql=0x19a48f00) at /export/home/pb2/bu
+ild/sb_0-10180689-1378750483.08/rpm/BUILD/mysql-5.5.34/mysql-5.5.34/s
+ql-common/client.c:2272
#14 0x00002ae284d7632d in mysql_real_connect (mysql=0x19a48f00, host=0
+x199e6b30 "10.30.30.73", user=0x19670e10 "<foo>", passwd=0x19670e30 "
+<foobar>", db=0x199e6af0 "<bar>", port=3306,
    unix_socket=0x0, client_flag=131074) at /export/home/pb2/build/sb_
+0-10180689-1378750483.08/rpm/BUILD/mysql-5.5.34/mysql-5.5.34/sql-comm
+on/client.c:3436
#15 0x00002ae284b2bfd5 in mysql_dr_connect (dbh=<value optimized out>,
+ sock=0x19a48f00, mysql_socket=0x0, host=0x199e6b30 "10.30.30.73", po
+rt=0x14edc368 "\b\243\016\210\342*",
    user=<value optimized out>, password=<value optimized out>, dbname
+=0x199e6af0 "<bar>", imp_dbh=0x193e98b0) at dbdimp.c:1892
#16 0x00002ae284b2d188 in my_login (my_perl=0x14772010, dbh=0x19696110
+, imp_dbh=0x193e98b0) at dbdimp.c:2004
#17 0x00002ae284b2e372 in mysql_db_login (dbh=0x19696110, imp_dbh=0x19
+3e98b0, dbname=0x2ae288132220 "<bar>:10.30.30.73:3306", user=0x2ae288
+132240 "<foo>", password=0x2ae288132260 "<foobar>")
    at dbdimp.c:2055
#18 0x00002ae284b35a77 in XS_DBD__mysql__db__login (my_perl=0x14772010
+, cv=<value optimized out>) at mysql.xsi:104
#19 0x000000000048906f in Perl_pp_entersub ()
#20 0x000000000048765e in Perl_runops_standard ()
#21 0x000000000043097f in Perl_call_sv ()
#22 0x00002ae27d64b5f1 in XS_DBI_dispatch () from /usr/local/lib/perl5
+/site_perl/5.10.1/x86_64-linux-thread-multi/auto/DBI/DBI.so
#23 0x000000000048906f in Perl_pp_entersub ()
#24 0x000000000048765e in Perl_runops_standard ()
#25 0x00000000004314bc in perl_run ()
#26 0x000000000041d4ad in main ()
(gdb)

[download]

Of the above, line 8 and 15 seem suspicious to me, but I can use some wisdom here.

In reply to Help debugging a random seg fault by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.