Does anyone have suggestions for this puzzler?

I construct a hash of hashes with DBI, then I walk a filesystem and process some files based on hash values. This has worked fine for months.

Recently, I added another hash layer from a new query. This worked; then gave a segmentation fault one day; then continued working. I can reproduce the error by fudging the hash so that the same set of files is processed.

The error occurs when an external java program is called to process a file (but not the first file!) If I print the java command, I can execute it at the command line. And no specific file causes the error - I can process each file individually, even in various combinations.

I dumped the hash and it looks fine. In the sample below, 'ABDQ' corresponds to a directory; 'wait' are files that shouldn't be processed yet; and 'done' are files that have already been processed. Any other files that are found in the directory are processed. There are about 300 similar entries, and the entire hash has ~50K keys. The new hash layer ('wait') constitutes only ~400 keys. (I’ve worked with much larger hashes on the same system.)

          'ABDQ' => {
                       'id'   => '212197'
                       'wait' => {
                                   'p.txt' => 1,
                                   'q.txt' => 1,
                                 },
                       'done' => {
                                   'a.txt' => 1,
                                   'b.txt' => 1,
                                   'c.txt' => 1,
                                   'd.txt' => 1,
                                   'e.txt' => 1
                                 },
                    },
          ...
[download]

Playing around, I’ve found that:

• If I delete corresponding hash entries as I process a directory's files, I don't get the error.
• If I don't add the new info to the hash, I don't get the error.
• If I add the new info to a separate hash, I still get the error; but if in the separate hash I swap the keys and values, I don't get the error!

 

Also, hoping to get more information, I added the following line. With it, the script hangs; without it, segfault.

$SIG{SEGV} = sub { require Carp; Carp::confess(@_) };
[download]

 

I'm using 5.8.0. I'm looking to try on a newer version.

Below are pstack and mdb::stack listings of the core. I’m reading up on how to dig deeper.

-----------------  lwp# 1 / thread# 1  --------------------
 ff14247c realfree (2e66ada8, ff1c2858, ff1bc008, ff1c27c8, 2e301a21, 
+369390) + 70
 ff142010 _malloc_unlocked (1408, 0, ff1bc008, 1408, 367f78, 2) + 24c
 ff141da8 malloc   (1408, 0, 0, 0, 21eec, ff199b94) + 20
 ff18f224 _findbuf (112630, 5, ff1bc008, ffffffff, 1400, 5) + 94
 ff18f0d8 _filbuf  (112630, 1, 0, ff1bc008, 40, 40) + 54
 ff1939c4 getc     (112630, ff1c3a54, ff1bfcd8, 112630, 0, 0) + bc
 000d87ec PerlIOStdio_read (0, ffbed67f, 1, d87d0, 0, 0) + 1c
 000d6cd8 Perl_PerlIO_read (122030, ffbed67f, 1, d8a20, 0, 0) + 2c
 000da77c PerlIO_getc (122030, 0, 0, 7efefeff, 54e1f0, 0) + c
 0008ae00 Perl_sv_gets (2a9528, 122030, 0, 0, 216fc, 40) + 574
 000aca90 Perl_pp_backtick (117034, 0, 112800, 1, 290, 0) + 208
 00078850 Perl_runops_standard (112400, 301a00, 112800, 114ff4, eb478,
+ 0) + 30
 00027848 S_run_body (1, 11250c, ffbef9c8, 0, 0, 0) + 150
 0002744c perl_run (0, 112800, 3, ffbefb04, 0, 0) + 90
 00024458 main     (0, ffbefb04, ffbefb14, 111fbc, 0, 0) + 94
 00024258 _start   (0, 0, 0, 0, 0, 0) + 5c
-----------------  lwp# 2 / thread# 2  --------------------
 ff19f204 _signotifywait (fef6c000, 0, 0, 0, ff1d18bc, fef6e000) + 8
 ff19a6a8 thr_errnop (0, 0, 0, 0, 0, 0) + 20
-----------------  lwp# 3  --------------------------------
 fef59300 private___lwp_cond_wait (4, fef6cd74, fef6c000, 0, 0, 4) + 8
 ff19ce00 _door_return (fef35cd8, fef4a380, 0, 0, 0, 0) + 68
-----------------  lwp# 4  --------------------------------
 ff19cda8 _door_return (4, fef6c000, fef6d678, 3, fef6c000, 1) + 10
 fef4a380 _lwp_start (fe20bd98, 0, 0, 0, 0, 0) + 18
 ff19a6a8 thr_errnop (0, 0, 0, 0, 0, 0) + 20
-----------------  lwp# 5  --------------------------------
 fef59300 private___lwp_cond_wait (4, fef6cd74, fef6c000, 3, fef6c000,
+ 1) + 8
 fef4a380 _lwp_start (fe109d98, 0, 0, 0, 0, 0) + 18
 ff19a6a8 thr_errnop (0, 0, 0, 0, 0, 0) + 20
--------------------------  thread# 3  --------------------
 fef4d9e0 _reap_wait (fef70988, 1e8fc, 0, fef6c000, 0, 0) + 38
 fef4d738 _reaper  (fef6ce08, fef72710, fef70988, fef6cde0, 1, fe40100
+0) + 38
 fef5b11c _thread_start (0, 0, 0, 0, 0, 0) + 40
[download]

mdb ::stack:

libc.so.1`realfree+0x70(2e66ada8, ff1c2858, ff1bc008, ff1c27c8, 2e301a
+21, 369390)
libc.so.1`_malloc_unlocked+0x24c(1408, 0, ff1bc008, 1408, 367f78, 2)
libc.so.1`malloc+0x20(1408, 0, 0, 0, 21eec, ff199b94)
libc.so.1`_findbuf+0x94(112630, 5, ff1bc008, ffffffff, 1400, 5)
libc.so.1`_filbuf+0x54(112630, 1, 0, ff1bc008, 40, 40)
libc.so.1`getc+0xbc(112630, ff1c3a54, ff1bfcd8, 112630, 0, 0)
PerlIOStdio_read+0x1c(0, ffbed67f, 1, d87d0, 0, 0)
Perl_PerlIO_read+0x2c(122030, ffbed67f, 1, d8a20, 0, 0)
PerlIO_getc+0xc(122030, 0, 0, 7efefeff, 54e1f0, 0)
Perl_sv_gets+0x574(2a9528, 122030, 0, 0, 216fc, 40)
Perl_pp_backtick+0x208(117034, 0, 112800, 1, 290, 0)
Perl_runops_standard+0x30(112400, 301a00, 112800, 114ff4, eb478, 0)
S_run_body+0x150(1, 11250c, ffbef9c8, 0, 0, 0)
perl_run+0x90(0, 112800, 3, ffbefb04, 0, 0)
main+0x94(0, ffbefb04, ffbefb14, 111fbc, 0, 0)
_start+0x5c(0, 0, 0, 0, 0, 0)
[download]