Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Monks,

I am maintaining a perl/xs application on windows and we've had a bug report with the error:

"Too many open files"

Digging into the report the user is opening lots of files with the application (many thousands) so it seems that we're not closing the file handle(s) within the application.

I've gone through the code and searched for all places that we open a file in perl (and within XS) and they all seemed to be paired with a close.

I'm stuck:) I don't know what to do next? Is it possible that the error is related to something else? Any tips on how to debug this?

Regards,

Replies are listed 'Best First'.
Re: Too many open files error on windows
by BrowserUk (Patriarch) on Jan 14, 2014 at 11:08 UTC
    Is it possible that the error is related to something else?

    No.

    By default, Perl can open 2048 files concurrently. Excluding the 3 standard files (stdin,stdout,stderr) that leaves 2045 for program use:

    perl -E"say(),open $fh[$_], '>', qq[file$_] or die $! for 1 .. 1e5" 1 2 3 ... 2040 2041 2042 2043 2044 2045 2046 Too many open files at -e line 1.
    Any tips on how to debug this?

    You could modify all the open and close lines something like this:

    open $fh, .... or die ...; close $fh;

    To

    open $fh, ... and printf( LOG "%s(%u) opening %u\n", __FILE__, __LINE_ +_, fileno( $fh ) ) or die ...; printf( LOG "%s(%u) closing %u\n", __FILE__, __LINE__, fileno( $fh ) ) + and close $fh;

    Once the program fails it should be simple to find which opens do not have or are not reaching their associated close statements.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Too many open files error on windows
by marto (Cardinal) on Jan 14, 2014 at 10:43 UTC

    Have you checked OS (you don't say which version) documentaiton for limits on such things? Has the user hit a resource limit? Perhaps there is something wrong with your code, or it's not doing what you think. Have you attempted to debug your code while recreating the problem? Tutorials -> Debugging and Optimization.

      Thanks marto for the reply.

      The application is used for batch processing, so the user had 2700 objects (files) to process, and it's the repeated processing of these objects that cause the failure. The user is using windows 7.

      I'm almost certain that our code is wrong, but am not sure where to start debugging. Creating a repeatable case is difficult as the objects that the user are using are sensitive. I've tested the app at the high level (process monitor and explorer) and I can't see a file handle leak so I was hoping to do something at the perl level?

Re: Too many open files error on windows
by nikosv (Deacon) on Jan 14, 2014 at 10:50 UTC
    you could use sysinternals process explorer, to check on all kinds of handles used by the application, just look for the File ones, so you can check if and which ones are still open.
Re: Too many open files error on windows
by GrandFather (Saint) on Jan 14, 2014 at 20:59 UTC

    Probably you already thought of this, but just in case: even if the opens and closes are paired you will run into trouble if you have too many files open concurrently - you open new files while old files are still open. A subtle way that could happen is if you have recursion (perhaps by accident) so check sub calls for that possibility.

    True laziness is hard work
      Thanks to all who replied, your suggestions were much appreciated.

      For any one who reads this thread in the future:

      The user's data (once we got a copy) was causing recoverable failures which left file handles open (in XS code).

Re: Too many open files error on windows
by Anonymous Monk on Jan 14, 2014 at 11:04 UTC

    improve the bug reporting to detect bugs in your bug reporting ?

    try to duplicate the problem?

    $ perl -le " print $!=$_ for 23 .. 24 " Too many open files in system Too many open files