Thanks for the $|. It definitely helps put things in perspective.
I maintain that the other processes are, indeed, waiting behind the looping process to be killed.
Here's why.
All I did was change the TIMEOUT constant from 120 to 30.
Here's the output at 120 seconds:
Fri May 15 11:29:37 2015 Started ==> test1.sh
Fri May 15 11:29:37 2015 Started ==> test2.sh
Fri May 15 11:29:37 2015 Started ==> test3.sh
Fri May 15 11:29:37 CDT 2015 I am running test1.sh
Fri May 15 11:29:37 CDT 2015 I am running test2.sh
Fri May 15 11:29:38 CDT 2015 I am running test3.sh
Fri May 15 11:29:37 2015 Ended ==> test3.sh
Fri May 15 11:29:37 2015 Started ==> test4.sh
Fri May 15 11:29:39 CDT 2015 I am running test4.sh
Fri May 15 11:29:37 2015 Ended ==> test2.sh
Fri May 15 11:29:37 2015 Ended ==> test1.sh
Fri May 15 11:29:37 2015 Started ==> test5.sh
Fri May 15 11:29:40 CDT 2015 I am running test5.sh
Fri May 15 11:29:37 2015 Ended ==> test5.sh
Fri May 15 11:29:49 CDT 2015 I am running test4.sh
Fri May 15 11:29:59 CDT 2015 I am running test4.sh
Fri May 15 11:30:09 CDT 2015 I am running test4.sh
Fri May 15 11:30:19 CDT 2015 I am running test4.sh
Fri May 15 11:30:29 CDT 2015 I am running test4.sh
Fri May 15 11:30:39 CDT 2015 I am running test4.sh
Fri May 15 11:30:49 CDT 2015 I am running test4.sh
Fri May 15 11:30:59 CDT 2015 I am running test4.sh
Fri May 15 11:31:09 CDT 2015 I am running test4.sh
Fri May 15 11:31:19 CDT 2015 I am running test4.sh
Fri May 15 11:31:29 CDT 2015 I am running test4.sh
Fri May 15 11:31:39 CDT 2015 I am running test4.sh
Fri May 15 11:29:37 2015 Ended ==> test4.sh
Fri May 15 11:31:51 2015 Started ==> test1.sh
Fri May 15 11:31:51 2015 Started ==> test2.sh
Fri May 15 11:31:51 2015 Started ==> test3.sh
Fri May 15 11:31:52 CDT 2015 I am running test1.sh
Fri May 15 11:31:52 CDT 2015 I am running test2.sh
Fri May 15 11:31:52 CDT 2015 I am running test3.sh
Fri May 15 11:31:51 2015 Ended ==> test3.sh
Fri May 15 11:31:51 2015 Ended ==> test1.sh
Fri May 15 11:31:51 2015 Ended ==> test2.sh
Here's the output at 30 seconds:
Fri May 15 12:58:08 2015 Started ==> test1.sh
Fri May 15 12:58:08 2015 Started ==> test2.sh
Fri May 15 12:58:08 2015 Started ==> test3.sh
Fri May 15 12:58:08 CDT 2015 I am running test1.sh
Fri May 15 12:58:09 CDT 2015 I am running test2.sh
Fri May 15 12:58:09 CDT 2015 I am running test3.sh
Fri May 15 12:58:08 2015 Ended ==> test2.sh
Fri May 15 12:58:08 2015 Ended ==> test1.sh
Fri May 15 12:58:08 2015 Ended ==> test3.sh
Fri May 15 12:58:08 2015 Started ==> test4.sh
Fri May 15 12:58:08 2015 Started ==> test5.sh
Fri May 15 12:58:11 CDT 2015 I am running test4.sh
Fri May 15 12:58:11 CDT 2015 I am running test5.sh
Fri May 15 12:58:08 2015 Ended ==> test5.sh
Fri May 15 12:58:21 CDT 2015 I am running test4.sh
Fri May 15 12:58:31 CDT 2015 I am running test4.sh
Fri May 15 12:58:41 CDT 2015 I am running test4.sh
Fri May 15 12:58:08 2015 Ended ==> test4.sh
Fri May 15 12:58:53 2015 Started ==> test1.sh
Fri May 15 12:58:53 2015 Started ==> test2.sh
Fri May 15 12:58:53 2015 Started ==> test3.sh
Fri May 15 12:58:53 CDT 2015 I am running test1.sh
Fri May 15 12:58:53 CDT 2015 I am running test2.sh
Fri May 15 12:58:53 CDT 2015 I am running test3.sh
Fri May 15 12:58:53 2015 Ended ==> test2.sh
Fri May 15 12:58:53 2015 Ended ==> test1.sh
Fri May 15 12:58:53 2015 Ended ==> test3.sh
If it were true that the other two forkable processes were processing work as soon as it's available, why does it take over 2 minutes for test1-3,5 to run when the test4 timeout is 120 but only a bit over 30 secs when it's set to 30?
This tells me that everything is waiting for the last process to finish in order to satisfy the wait_all_children.
I would expect test4 to take TIMEOUT seconds to complete, but I would also expect test1-3,5 to run as soon as a forkable process is available (i.e., after the 10 second sleep in the while loop).
What am I missing here?