Parallel ForkManager problem: parent dies early before all the child finishes

dayem98 has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Parallel ForkManager problem: parent dies early before all the child finishes by ikegami (Patriarch) on Aug 18, 2011 at 18:09 UTC
What error message do you get? What exit code do you get? Do you reach the line after `Wait_all_children`? Do you have signal handlers in the parent?	[reply] [d/l]
Re^2: Parallel ForkManager problem: parent dies early before all the child finishes by dayem98 (Initiate) on Aug 19, 2011 at 13:33 UTC
I have just checked and found out that the parent process is dying on `wait_all_children`	[reply] [d/l]
Re^2: Parallel ForkManager problem: parent dies early before all the child finishes by dayem98 (Initiate) on Aug 19, 2011 at 09:22 UTC
Basically the perl cgi is called upon from an html form and it does some expensive database operations based on batch file input. The program is working fine with relatively smaller input (say 10000 input sequence). The children are doing their bit of work properly and upon termination the main process starts again and do the final bit of calculation. Problem happens for larger batch files. I tested with 50000 input sequences and found out that all the children are created properly and finish their job accordingly. but during their run, parent dies abruptly. The result is the html page that is waiting for the result from the cgi is waiting forever, because the original process is no longer active. Not getting any error message, the main process simply dies while some children are still active. I didnt use any signal handler in the parent. For the larger bach processing, the main cgi does not reach the line after `wait_all_children`, but for the smaller batch processing, it indeed does and finishes the job.	[reply] [d/l]
Re^3: Parallel ForkManager problem: parent dies early before all the child finishes by zentara (Cardinal) on Aug 19, 2011 at 14:22 UTC
That is probably the Apache web server timing out your process. A http server does not like to wait around on long processes, there are ways around it. See Managing a long running server side process using CGI. You might check the httpd error logs and see if you see an error about timeout, or google for expired: ap_content_length_filter: apr_bucket_read() failed. . A most often recommended read for this problem is Merlyn's watching long processes thru cgi I'm not really a human, but I play one on earth. Old Perl Programmer Haiku ................... flash japh	[reply]
Re^4: Parallel ForkManager problem: parent dies early before all the child finishes by dayem98 (Initiate) on Aug 19, 2011 at 14:55 UTC
Re^5: Parallel ForkManager problem: parent dies early before all the child finishes by zentara (Cardinal) on Aug 19, 2011 at 17:56 UTC
Re^3: Parallel ForkManager problem: parent dies early before all the child finishes by ikegami (Patriarch) on Aug 22, 2011 at 22:17 UTC
You missed the most important one. What exit code do you get?	[reply]
Re: Parallel ForkManager problem: parent dies early before all the child finishes by zentara (Cardinal) on Aug 18, 2011 at 16:52 UTC
A guess just from glancing at your code, your (MAX_PROCESSES) is ($tc-1). When you go into your loop, your upper limit is $tc. It's conceivable, your MAX_PROCESSES is being hit. If a count starts at 0, you might be 2 off from 10. I'm not really a human, but I play one on earth. Old Perl Programmer Haiku ................... flash japh	[reply]
Re^2: Parallel ForkManager problem: parent dies early before all the child finishes by delirium (Chaplain) on Aug 18, 2011 at 17:03 UTC
I don't think ForkManager crashes if you go over the limit; I think it waits. It's possible there is a system limit on the number of process that you're going over.	[reply]
Re: Parallel ForkManager problem: parent dies early before all the child finishes by Mr. Muskrat (Canon) on Aug 19, 2011 at 18:10 UTC
Do you really want `$dbh->{InactiveDestroy} = 1` there? In that context, wouldn't it make more sense to use `$child_dbh->{InactiveDestroy} = 1`?	[reply] [d/l] [select]
Re^2: Parallel ForkManager problem: parent dies early before all the child finishes by runrig (Abbot) on Aug 19, 2011 at 18:38 UTC
No, it's the parent handle that you don't want to disconnect when the child process ends, and that's what the attribute does, it prevents disconnect upon destruction. Although I now notice in the docs an 'AutoInactiveDestroy' attribute, which looks more convenient.	[reply]
Re^3: Parallel ForkManager problem: parent dies early before all the child finishes by Mr. Muskrat (Canon) on Aug 22, 2011 at 17:56 UTC
Even the DBI docs say that you probably want to set InactiveDestroy in the child (although it says so in the section for AutoInactiveDestroy.) Update: Okay, I see what I did there. LOL	[reply]
Re: Parallel ForkManager problem: parent dies early before all the child finishes by Anonymous Monk on Aug 21, 2011 at 15:15 UTC
Yeah, Apache is probably killing-off the parent. In any case, if the operation is "expensive," the web server process ought not be the one who's doing it ... or orchestrating it. Web pages should just be a user-interface. If you have an expensive or long-running thing to do, a batch manager should be doing it, separate from Apache or from anyone else.	[reply]