in reply to Threads memory consumption is infinite

I'm just brainstorming here, from other threads experience. A thread gets a copy of the parent when it gets spawned, and this is the cause of all sorts of thread safety difficulties,

Now just glancing at your pseudocode, you are first creating up to 20 parser threads BEFORE you spawn the $mysql_thread, so the $mysql_thread gets a copy of all of it. Possibly you are getting recursion in parser thread creation also? What sequence would add up to 170? Does the second parser thread get a copy of the first, etc.

Maybe try to spawn your $mysql_thread BEFORE you create your 20 parser threads? Also can you do 

$mysql_thread->kill(’SIGUSR1’);
undef $mysql_thread;

[download]
possibly $threads->exit can be used in the thread to ensure it returns, so it can close itself up.

But like BrowserUk suggests, your best bet is to simplify it down to a testable example, without mysql involved, and see how it behaves.

I'm not really a human, but I play one on earth CandyGram for Mongo

Replies are listed 'Best First'.
Re^2: Threads memory consumption is infinite
by Godsrock37 (Sexton) on Jun 10, 2008 at 16:57 UTC

    watching task manager i see that the mysql threads do successfully die and there are never more than 25 threads or so created (20 parsers, 1 main, and 2-4 mysql which fluctuate depending on performance)

    i told it to die after about 15 minutes (all that it can last) and as each thread exits it frees up 300-400MB approximately which tells me that each parser thread is accumulating the memory, and not the mysql threads or even the main thread that spawns the parsers... hopefully that was a coherent thought

[reply]
      hopefully that was a coherent thought

      heh heh :-)

      I'm not really a human, but I play one on earth CandyGram for Mongo
[reply]

        I'm looking over my algorithm and you're comment and I'm wondering if it would be better to do the following:

        sub main_loop {
  while(
    schedule_count()
    and $hit_count < $hit_limit       #could be commented out
    and time() < $expiration
    and ! $QUIT_NOW
  ) {
    #yield();
    my $parser_thread1 = async { process_url( next_scheduled_url() );}
+;
  }
  return;
}

        [download]

        ... rather than have the 20 static threads that loop forever, each thread would exit like the mysql threads which seem to be working fine

[reply]
[d/l]

        thats what i said :)

[reply]

In Section Seekers of Perl Wisdom