plagent has asked for the wisdom of the Perl Monks concerning the following question:

I used Parallel::ForkManager to scan the motifs in DNA string pairs. The script is the following:
my $multi_cpu_num = 13; my @ciona_pwm_array = getCionaPWM(); my $output_temp = 'temp.ciona.temp'; my $output_temp_handle = FileHandle->new(">output_temp"); my $output_handle = FileHandle->new(">ciona.matrix.result"); my $result_buffer = undef; foreach my $id (1..100) { my $seq_1 = 'gatctatcgtagcagcta'; my $seq_2 = 'agtctacatgctacgatg'; my $pm = Parallel::ForkManager->new( $multi_cpu_num ); PWM: for my $pwm (@ciona_pwm_array) { $pm->start and next PWM; my $count = count_pwm_num($pwm,$seq_1,$seq_2); if($count != 0) { $output_temp_handle->print($id); $output_temp_handle->print("\t$pwm\t1\n"); } else { $output_temp_handle->print($id); $output_temp_handle->print("\t$pwm\t0\n"); } $pm->finish; } $pm->wait_all_children; } $output_temp_handle->close; # catch the Parallel::ForkManager output # the temporary output looks like that # a A 1 # b B 0 # c C 1 # a B 1 # in real code , I used 1, 2,3, instead of a, b, c ,d. my $reopen_handle = FileHandle->new($output_temp); while(my $line = $reopen_handle->getline) { chomp $line; my @buffer = split/\t/,$line; $result_buffer->{$buffer[0]}->{$buffer[1]} = $buffer[2]; } $reopen_handle->close; unlink 'temp.ciona.temp'; # print output_header_information :row 1 in program result table; $output_handle->print("id"); foreach my $pwm (@ciona_pwm_array) { $output_handle->print("\t$pwm"); } $output_handle->print("\n"); # print the content of program result table foreach my $id (1..100}) { $output_handle->print("$id"); foreach my $pwm (@ciona_pwm_array) { my $count = $result_buffer->{$id}->{$pwm}; $output_handle->print("\t$count"); } $output_handle->print("\n"); } $output_handle->close; sub ciona_pwm_num {} sub getCionaPWM {] exit(0);
There are three parameters for the function count_pwm_num. The first one is $pwm, which is from list @ciona_pwm_array and is the motif feature. The second and third are DNA strings. The function is to calculate the motif number (how many?)using motif feature $pwm in the DNA strings. The aim of the function count_pwm_num is to report the motif number from both strings. The following table is the report from the program. However, I calculated the same DNA strings 100 times ( the first for-loop) and used same motif list (@ciona_pwm_array, the second for-loop), there is always missed motif(s) in the report. For example, row a (first row), the program reported all six motif number from A-F. row b missed the number of the motif A (because they were same DNA strings, the output should keep the same). row c missed the number of motif B.row d missed the number for motif A and B. It seemed the missed motif numbers were randomly selected. So my question is: why count_pwm_num randomly skip some $pwm and do not perform the function of count_pwm_num?

program result table:

id	A	B	C	D	E	F
a	1	1	0	1	1	0
b		1	0	1	1	0
c	1		0	1	1	0
d			0	1	1	0
e	1	1	0	1	1	0
..

Expected result

id	A	B	C	D	E	F
a	1	1	0	1	1	0
b	1	1	0	1	1	0
c	1	1	0	1	1	0
d	1	1	0	1	1	0
e	1	1	0	1	1	0
..

Replies are listed 'Best First'.
Re: missing values when using Parallel::ForkManager
by kcott (Archbishop) on Jul 03, 2015 at 11:07 UTC

    G'day plagent,

    You haven't shown the code that actually generates the report. That makes it rather difficult to troubleshoot.

    The actual output you show has staggered columns. This suggests to me that data for various columns is either whitespace or empty strings. The latter could be the result of an uninitialized value being treated as "". The warnings pragma will alert you to this.

    Other potential issues include:

    • $id is set on each iteration of the outer loop but never used.
    • $count is set on each iteration of the inner loop but never used.
    • You haven't shown any declaration for $count. The strict pragma will alert you to undeclared variables.
    • You're calling count_pwm_num() in list context (($count) = ...) which seems inappropriate for a function which returns a number. Obviously, as you haven't shown the code for count_pwm_num(), that's just a guess on my part.

    If one or more of those suggestions and guesses lead to a solution, that's great. If not, you'll need to post more code for us to help you further. Also include any output from the two pragmata I mentioned: given they're both lexically scoped, it's important to show where in your code that you've included them.

    -- Ken

      Thanks, Ken. I updated the source code, and hope this light version will explain my trouble. In fact, there was complaint from running the above code. It is "Use of uninitialized value $count". The code of this line is
      my $count = $result_buffer->{$id}->{$pwm};
      The missing value here indicates that this piece of code
      my $count = count_pwm_num($pwm,$seq_1,$seq_2);
      did not perform its function when parameters were transferred here. $pwm is from list @ciona_pwm_array. Because after executing those lines, the result will catch either 1 or 0,
      if($count != 0) { $output_temp_handle->print($id); $output_temp_handle->print("\t$pwm\t1\n"); } else { $output_temp_handle->print($id); $output_temp_handle->print("\t$pwm\t0\n"); }
      However, as shown in result table, several values were missed from certain rows, (row c missed value 1 in column B, row b missed value 1 in column A. The missing values have no specific pattern, and it seems randomly lost. I doubted for some reason it did not execute this code
      $pm->start and next PWM;
      But I do not know why. Thanks.
        I wrote a mini code for this problem which replicates my puzzle.
        #!/usr/bin/perl use strict; use Parallel::ForkManager; my $pm = Parallel::ForkManager->new(10); foreach my $j (1..100) { LINKS: for my $i (1..10) { $pm->start and next LINKS; my $num = count($i); print $num,"\t"; $pm->finish; } print "\n"; $pm->wait_all_children; } sub count { my $num = shift; return $num + 1; }
        I expected each row should generate ten numbers from 2 to 11. However, several rows will miss values. Even worse, several rows output 11 numbers instead of 10 numbers, and have replicate, for example, 10 appear twice. Why?