in reply to Re: Logical Conundrum
in thread Logical Conundrum

To help clarify for mcogan1966:

From what he said in the cb, he comes across as wanting to do all the data munging at once, but print out the results together (unmixed) at the end of the processing. So it's kind of one-by-one and parallel combined. Parallel computing , one-by-one results. Which is an ideal use of flock(), for which he seems to be having some odd issues. He hasn't posted code to show us what might be going wrong, but it seems he is passing the locked handle to forked children. Which is just far too wrong :)

Replies are listed 'Best First'.
Re^3: Logical Conundrum (bugs?)
by tye (Sage) on Dec 01, 2003 at 19:20 UTC

    Passing the file-handle-to-be-locked to children avoids one race condition and allows the parent to unlink the temporary lock file as soon as it is created (improving security).

    Unfortuantely, BSD-based flock has different ideas about such things, allowing different processes to share the lock (IIRC). This makes some things easier and some things harder. Perhaps letting the children inherit the handle-to-be-locked but having each child dup() it (see open) instead of locking it directly would give you the best of both worlds (I think and hope that BSD doesn't go so far as to prevent that from working).

    Since it is output from Perl that is trying to be serialized (in the sense of synchronization not in the sense of marshaling), one likely bug would be: 

        lock(...);
    print ...;
    unlock(...);

    [download]
    because the output doesn't happen when print is called but when the buffer is flushed. Simply adding $|= 1; should fix that in this simple case. For more complex cases, it might be easier to do: 
        lock(...);
    print PIPE ...;
    close PIPE;
    unlock(...);

    [download]

                    - tye
[reply]
[d/l]
[select]
Re: Logical Conundrum
by Abigail-II (Bishop) on Dec 01, 2003 at 19:36 UTC
    As tilly pointed out, it's doable with locks, but since the semantics of locks after a fork differs from system to system, you first need to know the system before you can present a solution.

    But there's a different solution. Instead of fork/exec, with the children doing the prints, open *pipes* to the children, and have the children write to the pipes. The the parent can decide in which order the results will be printed. Beware, if there's a lot of data to be printed, of which a lot is available long before the end of the life of the child, it might not work to well.

    Abigail

[reply]
      Actually, that was what the first version did, and it didn't work to well either. Hence the fork/system calls. Same problem, but things seemd to move faster. Not necessarily such a good thing, but if I didn't generate any new problem, I'll keep it that way, for now.

      My scratchpad has some of the code I'm trying to muddle through. Maybe that'll offer some ideas.

[reply]

In Section Seekers of Perl Wisdom