rmahin has asked for the wisdom of the Perl Monks concerning the following question:

Hi PerlMonks,

Been working at this one while, and haven't had much luck searching. I have boiled down my code to this little snippet which highlights my issue. Essentially I have threads that pipe the output of other programs to a filehandle but things get weird when I try to remove the files I write to. This is not a problem on any unix machines as far as I can tell but on every version of Windows I have tried the problem persists.

#! perl -l use strict; use warnings; use threads; $|++; sub thread{ my $tid = threads->tid; my $fn = "file" . $tid; open(my $fh, ">", $fn) or die "Couldnt open filehandle $!"; my $pid = open(my $fs, "-|", "perl test.pl") or die "Couldnt open +process: $!"; print $fh $_ while <$fs>; close($fs); close($fh); } my $thr1 = threads->create(\&thread); sleep 1; unlink("file1") or print "Cant remove file1 because $!"; my $thr2 = threads->create(\&thread); sleep 1; unlink("file2") or print "Cant remove file2 because $!"; $thr1->join(); unlink("file1") or print "Still cant remove file1 because $!"; $thr2->join(); unlink("file1") or print "This isnt printed $!"; unlink("file2") or print "This isnt printed $!";

And test.pl is pretty trivial but here ya go. (Yes, that's all that's in there)

sleep 5;

The output is:

Cant remove file1 because Permission denied
Cant remove file2 because Permission denied
Still cant remove file1 because Permission denied

So my question is how come I still cant remove file1 after that thread has closed the filehandle and the thread has been joined, why does thread 2's presence cause an issue? I'm assuming it has something to do with the fact the processes are both called "perl.exe" and are child processes of the main script but that doesn't help me in solving the issue...Using the utility Unlocker it does show that both of the test.pl processes are keeping file1 and file2 locked, but it doesn't make sense to me as to why that is, or how to fix it. Any help, you guys can give is most appreciated. Thanks!

  • Comment on Multithreaded script keeps files locked when reading output from multiple processes on Windows
  • Select or Download Code

Replies are listed 'Best First'.
Re: Multithreaded script keeps files locked when reading output from multiple processes on Windows (on win32 threads share STDIN/STDOUT/STDERR so close STDIN; close STDOUT;)
by Anonymous Monk on Jan 10, 2014 at 01:56 UTC

    On win32 threads share STDIN/STDOUT/STDERR, you get one of those per process, and when you do pipe-open, perl dup's each of STDIN/STDOUT/STDERR ...

    so something something fork emulation keeps the filehandles alive and in use and undeletable

    if you add close STDIN; close STDOUT; in sub thread, you'll get different output, like

    Filehandle STDIN reopened as only for output

    and Still cant remove file1 because No such file or directory instead of Permission denied
    This isnt printed No such file or directory

      Hi I appreciate your response! What you said definitely makes sense, however in that script regardless of where I close STDIN/STDOUT, I see the same output...Am I doing something wrong? Or could there be something else that is getting duplicated as well? Thanks again for the help!

Re: Multithreaded script keeps files locked when reading output from multiple processes on Windows
by BrowserUk (Patriarch) on Jan 10, 2014 at 20:26 UTC
    • First: The files aren't "locked".

      They are open without the FILE_SHARE_DELETE permission.

    • File handles are process global entities.

      This is true on all systems.

    • It is not "thread 2's presence cause an issue?" per se.

      The reason you cannot delete file1 even after thread 1 has ended is because when the 3rd copy of Perl.exe is spawned in thread 2, that process inherits all open file handles, including that of file 1.

      As another process has an open file handle without the FILE_SHARE_DELETE permission, the file cannot be deleted by the creating process until that other process terminates.

    If you run this modified version of your test code, you'll see that the main thread can delete file 1 whilst thread 2 is still running; but only after the process thread 2 spawns, terminates.

    The output:

    C:\test\junk>..\junk41 26: Cant remove file1 because Permission denied/The process cannot acc +ess the file because it is being used by another process 29: Cant remove file2 because Permission denied/The process cannot acc +ess the file because it is being used by another process [1] eof pipe [1] pipe closed [1] thread ending 32: Still cant remove file1 because Permission denied/The process cann +ot access the file because it is being used by another process 32: Still cant remove file1 because Permission denied/The process cann +ot access the file because it is being used by another process 32: Still cant remove file1 because Permission denied/The process cann +ot access the file because it is being used by another process 32: Still cant remove file1 because Permission denied/The process cann +ot access the file because it is being used by another process 32: Still cant remove file1 because Permission denied/The process cann +ot access the file because it is being used by another process 32: Still cant remove file1 because Permission denied/The process cann +ot access the file because it is being used by another process 32: Still cant remove file1 because Permission denied/The process cann +ot access the file because it is being used by another process 32: Still cant remove file1 because Permission denied/The process cann +ot access the file because it is being used by another process 32: Still cant remove file1 because Permission denied/The process cann +ot access the file because it is being used by another process 32: Still cant remove file1 because Permission denied/The process cann +ot access the file because it is being used by another process [2] eof pipe [2] pipe closed [2] thread ending 36: This isnt printed No such file or directory/The system cannot find + the file specified

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      Thanks for your explanation. Was able to work around this by opening files with the FILE_SHARE_DELETE permission as you using Win32API::File and it does seem to do the trick. Replaced the thread code with:

      my $tid = threads->tid; my $fn = "file" . $tid; my $winFileHandle = createFile( $fn, "w", "d" ) or die "$!/$^E"; OsFHandleOpen(*OUTPUT, $winFileHandle, "w") or die "$!/$^E"; my $pid = open(my $pipe, "-|", "perl test.pl"); print OUTPUT $_ while(<$pipe>); close(OUTPUT); close($pipe); CloseHandle($winFileHandle);

      I do however have another question, well more of a clarification. So in my actual usage, I am using a threadpool and a Thread::Queue and not starting/joining threads after one another. But, I'm just making sure I understand correctly, this does not matter because if I have for example:

      1. thread 1 open file1
      2. thread 2 start process A while file1 is still open
      3. thread 1 closes file1
      process A will still have a filehandle for file1 even though thread2 did absolutely nothing with the file because the spawned process inherits ALL of the the file handles currently in use by the main perl process at the time of opening the pipe? Sorry if this comes out more as just paraphrasing what you said, just wanting to make sure I understand fully :)

      Thanks for the help!

        process A will still have a filehandle for file1 even though thread2 did absolutely nothing with the file because the spawned process inherits ALL of the the file handles currently in use by the main perl process at the time of opening the pipe?

        That is correct, but overstates the conditions.

        Forget which thread does what. Indeed, forget threads entirely. (Even forget the file bit.)

        When one process spawns another process, the child process will inherit every open handle that has the bInheritHandle flag set in the Security Attributes structure supplied when that handle was created in the parent.

        At the Win32 API level, this can be controlled on a per (file) handle basis; but as POSIX doesn't have that concept, Perl doesn't provide access to it and thus uses a generic security descriptor that means everything gets inherited.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        Ignore this...created on accident