It's about 3000 files. I can't post all of it because it's on a classified network. All I can say is that it is mostly ascii file types, one .xls file, as well as one hex formatted file that I read through and do checks on. For each file I create a running log file with any errors of the file and at the end I insert a record into a database basically saying whether or not the file was bad. I guess if anything, I'll have to look to see if it's any of the modules I am using that could be a problem as well. Which are...

 
use threads(...);
use Thread:: Queue;
use File::Find;
use File::Basename;
use DBI;
use DBD::ODBC;
use Spreadsheet::ParseExcel;
use Switch;
use File::Find;

[download]

There is definitely a gradual increase in memory usage over time.

I don't have any references, which could lead to circular references that the perl garbage collector doesn't pick up.

Are there any tools I can use? Such as checking memory usage before and after a method call to ensure that the amount of used memory before and after the dealloc for that particular method is the same?

What happens in perl when two threads call on the same method? I assume each will just get their own copy, which will be fine since again, no shared resources.

In reply to Re^2: Multithreading leading to Out of Memory error by joemaniaci
in thread Multithreading leading to Out of Memory error by joemaniaci

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.