I have a script that uses threading. It has a shared variable ($ret) that is used to indicate if an error has occurred. All the threads check that variable so that work will stop in the event of a fatal error.

Unfortunately I cannot post the script. I'm not able to create a test script to replicate the issue since it is very intermittent and only seems to happen with long runs (over 4 hours). It doesn't use much memory, so it doesn't seem to be a memory leak.

Basically the script will start a new thread to handle writing output, and the main thread will start possibly millions of threads to process 500 records each as they are read using a semaphore to limit the number of concurrent threads. All threads check the value of the shared scalar. They can all modify the value, but didn't when these crashes occur.

>> /polaris_stg_root/dev/app/smartload/components/correctaddress_debug
+/correctaddress_debug.pl:11: $Devel::Trace::TRACE = 0;
SV = PVMG(0x10a6350) at 0xf68f68
  REFCNT = 5
  FLAGS = (PADMY,GMG,SMG,IOK,pIOK)
  IV = 0
  NV = 0
  PV = 0
  MAGIC = 0xd5fb30
    MG_VIRTUAL = 0x7f361a0c9320
    MG_TYPE = PERL_MAGIC_shared_scalar(n)
    MG_FLAGS = 0x30
      DUP
      LOCAL
    MG_PTR = 0xee8f08 ""
SV = PVMG(0x114ec30) at 0x114d558
  REFCNT = 5
  FLAGS = (PADMY,GMG,SMG,IOK,pIOK)
  IV = 0
  NV = 0
  PV = 0
  MAGIC = 0x1150a70
    MG_VIRTUAL = 0x7f361a0c9320
    MG_TYPE = PERL_MAGIC_shared_scalar(n)
    MG_FLAGS = 0x30
      DUP
      LOCAL
    MG_PTR = 0xee8f08 ""
Attempt to free unreferenced scalar: SV 0xee8f08, Perl interpreter: 0x
+ee6410.
>> /polaris_stg_root/dev/app/smartload/components/correctaddress_debug
+/correctaddress_debug.pl:884: exit($ret) if($ret == -1); # already fa
+iled, don't compare counts or print stats
panic: attempt to copy freed scalar ee8f08 to f68f68 at /polaris_stg_r
+oot/dev/app/smartload/components/correctaddress_debug/correctaddress_
+debug.pl line 884.
Attempt to free unreferenced scalar: SV 0xee8f68, Perl interpreter: 0x
+ee6410.
Attempt to free unreferenced scalar: SV 0xee8f08, Perl interpreter: 0x
+ee6410.
[download]

One dump is the $ret variable from before the output thread returns. The other is from the main thread after all the input is read. I should have dumped it after the output thread is joined and will if I can make it crash again. The main thread does not have any references to $ret between dumping it and line 884.

UPDATE

I got a dump after the output thread is joined, and the refcount is still 5. There are no references to $ret until it crashes, so it seems the memory is freed even though the refcount is 5?

>> /polaris_stg_root/dev/data/QAS_TEST/correctaddress_debug.pl:11: $De
+vel::Trace::TRACE = 0;
Main thread before output thread finishes:
SV = PVMG(0x19555c0) at 0x1822318
  REFCNT = 5
  FLAGS = (PADMY,GMG,SMG,IOK,pIOK)
  IV = 0
  NV = 0
  PV = 0
  MAGIC = 0x190b330
    MG_VIRTUAL = 0x7fe80f9de320
    MG_TYPE = PERL_MAGIC_shared_scalar(n)
    MG_FLAGS = 0x30
      DUP
      LOCAL
    MG_PTR = 0x17a7b68 ""
Output thread before returning:
SV = PVMG(0x19f2730) at 0x1a3d280
  REFCNT = 5
  FLAGS = (PADMY,GMG,SMG,IOK,pIOK)
  IV = 0
  NV = 0
  PV = 0
  MAGIC = 0x1a3e2e0
    MG_VIRTUAL = 0x7fe80f9de320
    MG_TYPE = PERL_MAGIC_shared_scalar(n)
    MG_FLAGS = 0x30
      DUP
      LOCAL
    MG_PTR = 0x17a7b68 ""
Main thread after output thread finishes:
SV = PVMG(0x19555c0) at 0x1822318
  REFCNT = 5
  FLAGS = (PADMY,GMG,SMG,IOK,pIOK)
  IV = 0
  NV = 0
  PV = 0
  MAGIC = 0x190b330
    MG_VIRTUAL = 0x7fe80f9de320
    MG_TYPE = PERL_MAGIC_shared_scalar(n)
    MG_FLAGS = 0x30
      DUP
      LOCAL
    MG_PTR = 0x17a7b68 ""
>> /polaris_stg_root/dev/data/QAS_TEST/correctaddress_debug.pl:884: ex
+it($ret) if($ret == -1); # already failed, don't compare counts or pr
+int stats
panic: attempt to copy freed scalar 17a7b68 to 1822318 at /polaris_stg
+_root/dev/data/QAS_TEST/correctaddress_debug.pl line 884.
Attempt to free unreferenced scalar: SV 0x17a7b68, Perl interpreter: 0
+x1775870.
[download]

UPDATE 2

I changed the script to not use the global $ret variable (shared or not) from the input, output, or worker threads. This seems to keep my script from crashing. I may use marioroy's MCE approach in a future version.