I'm having an issue keeping the memory usage of a web spider i wrote down. It runs fine without any issues or bugs except that after a few hours the computer runs out of memory and i get an error that there isnt enough available memory to allocate what i need. I'm running on Windows XP with 4 GB of RAM, watching task manager shows that it steadily climbs

i'm using threading to keep the program running at a decent speed (20 threads) and i spawn threads that detach to do mysql statements as well

i've looked everywhere for help on this issue including here, and seen that there were bug fixes in past versions of perl but it seems like it is a persistent issue.

im using perl 5.10 and threads 1.67

i've told threads to make the stack size 4096 and i've even explicitly declared the detached mysql threads undef when they're done. Final output is as follows:

Out of memory!
Callback called exit at C:/perl/lib/HTML/Element.pm line 234.
Callback called exit at C:/perl/lib/HTML/Element.pm line 234.
Perl exited with active threads:
    19 running and unjoined
    0 finished and unjoined
    2 running and detached

[download]

the full program is 700 lines so i've only posted how i spawn the threads

use threads ('stack_size' => 4096);
use threads::shared;
use threads qw(yield);
use Thread::Queue;
my $sched = new Thread::Queue;

my $parser_thread1 = async { main_loop();};
my $parser_thread2 = async { main_loop();};
my $parser_thread3 = async { main_loop();};
#goes to 20, not elegant but it works

$parser_thread1->join;
$parser_thread2->join;
$parser_thread3->join;
#goes to 20 as well


sub main_loop {
  while(
    schedule_count()
    and $hit_count < $hit_limit       #could be commented out
    and time() < $expiration
    and ! $QUIT_NOW
  ) {
    yield();
    process_url( next_scheduled_url() );
  }
  return;
}


my $mysql_thread = threads->create('send_parts_to_db', $parts));
$mysql_thread->detach;
undef $mysql_thread; #as suggested by someone on perlmonks

[download]

As shown i use the same mysql_thread variable for all mysql insertions... but if i ever get an error i can have up to thread # 170+ Is that normal?

In reply to Threads memory consumption is infinite by Godsrock37

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.