in reply to Re: Threads memory consumption is infinite
in thread Threads memory consumption is infinite

watching task manager i see that the mysql threads do successfully die and there are never more than 25 threads or so created (20 parsers, 1 main, and 2-4 mysql which fluctuate depending on performance)

i told it to die after about 15 minutes (all that it can last) and as each thread exits it frees up 300-400MB approximately which tells me that each parser thread is accumulating the memory, and not the mysql threads or even the main thread that spawns the parsers... hopefully that was a coherent thought

  • Comment on Re^2: Threads memory consumption is infinite

Replies are listed 'Best First'.
Re^3: Threads memory consumption is infinite
by zentara (Cardinal) on Jun 10, 2008 at 17:03 UTC
    hopefully that was a coherent thought

    heh heh :-)

    I'm not really a human, but I play one on earth CandyGram for Mongo
[reply]

      I'm looking over my algorithm and you're comment and I'm wondering if it would be better to do the following:

      sub main_loop {
  while(
    schedule_count()
    and $hit_count < $hit_limit       #could be commented out
    and time() < $expiration
    and ! $QUIT_NOW
  ) {
    #yield();
    my $parser_thread1 = async { process_url( next_scheduled_url() );}
+;
  }
  return;
}

      [download]

      ... rather than have the 20 static threads that loop forever, each thread would exit like the mysql threads which seem to be working fine

[reply]
[d/l]

      thats what i said :)

[reply]

In Section Seekers of Perl Wisdom