But anyway, I don't understand what ur trying to say about closing them. It's not that I want fewer handles open at one time, it's that I want them to close and completely go away between invocations of that subroutine. I don't see how operating on the handles in two loop structures makes any difference to their persistence. Let's say I got rid of the for() loop and manually opened a number of handles as variables in the @pings array. Ur saying that would make a difference? The handles are all opened in the context of a subroutine and operated on solely within that same subroutine. Asked another way, what situation would make a file handle that just had close() called on it not release back to the OS?

Thanks.

Asked another way, what situation would make a file handle that just had close() called on it not release back to the OS?

close can fail, but if close succeeds, then there are four possibilities

• the monitoring program is lying (cached data...)
• the operating system is lying (think of chroot)
• you're looking at the wrong thing at the wrong time and reaching the wrong conclusion
• cosmic rays/gremlins
To me, it just looks like you're opening too many pings at one time

The subroutine is not shown entirely in your code. The point of declaration for your @pings array is not shown in your code, which means it is likely outside the scope shown. There is no evidence in your code that the file handles are being closed before hitting your ulimit. According to your code, there is a file handle opened for every host in your @hosts array. If your @hosts array has more members than your ulimit of file handles, then your code, as provided by you and nobody else, will attempt to open more file handles than your ulimit allows.

If you don't understand where you come across as cocky (I never said "smug", but close enough), I invite you to read the one curt reply you gave three people (in Re: old file descriptors not being cleaned up):

Thanks for the replies guys. Here are my answers. @anon: I monitor the process from another terminal. I run ps to get the pid and then do ls /proc/PID/fd and watch the list grow until it crashes at 256. @mr_mischief: I explicitly close them after sysread returns 0. That is in the elsif block where $len == 0. The close() returns true. Also the read is from a pipe to the ping command, not a file. @Khen1950fx: I'm not using seek. >>

The only potentially useful thing you said to me was to try to correct me about a pipe being seen as a file. I've got news for you: a pipe works using file handles and file opens and closes because although it is a special case, it is indeed treated as if it was a file. The same file descriptor limits apply. For all cases in which a piped file handle, a fifo, a device file (which was in my example in response to you), and a vanilla file on a mounted file system act the same, the distinction is irrelevant. Only the ways in which they differ matter.

Furthermore, the very problem I warned you about you glossed over in your rudely short and impersonal reply. I told you it's a good idea to close them where you open them. You specifically tell me that you are closing them, then point out where you are closing them. That place is not in the same loop in which you are opening them, which is what I told you was your most likely problem. I went on in my reply at Re: old file descriptors not being cleaned up to write you three example programs that illustrate the issue I warned you about. You made no mention of running them, perusing the code, comparing them to your situation, or contrasting why your code doesn't suffer from the same problem as the first example.

You were also entirely dismissive of both other people in the thread. The anonymous monk asked (in Re: old file descriptors not being cleaned up) what lsof said about the ownership of the open file handles. You ignored that. He or she also asked what the results of close calls are, but instead of saying in your reply or even asking what was meant you simply ignored that. Khen1950fx asked in Re: old file descriptors not being cleaned up about seek(), which you dismissed, then advised trying a seek on the file handle (which, again, is still what it is called even when there's a pipe involved). You dismissed the first mention of the seek function but ignored completely the second.

You exact description of your problem is:

The problem is that the file descriptors (or file handles) are sticking around even though the processes exit and I close()'d them. The script eventually dies with a "too many open files" message. The box is Solaris 10, Perl 5.12.2, and the ulimit is 256 handles. This can be raised to 1024 but it would still crash if I need to ping more than that. I've tried kill, readline, waitpid, set sig-child/pipe to ignore but they still build up.

If you're actually successfully closing the filehandles, then they won't be open. You example code does not show a test for making sure the close of the pipe is successful. This the anonymous monk asked abut already, but you ignored.

Since it's a pipe for reading from the chained command and you've already tested the length, there's little reason for it to fail. Still, to be really sure the filehandles are closed all the way down to the OS file descriptors you should test the close call. Once you can confirm you're actually successfully closing them when you think you are, you can move on to seeing if you're opening more than your ulimit allows before attempting to close them.

You could have spent just a little time reading the replies to your urgent (to you) request for help. You could have taken a bit of your time seeing how the replies we took our time to give you free of charge applied to your situation. You then could have replied to each person individually without completely ignoring certain points or refuting others without explanation. You could have refrained from belaboring points of terminology that make a distinction without a difference in an attempt at... what, exactly? General pedantry was the goal perhaps? Only you could answer that, and I really don't care anyway.

I'm only still interested in helping you for a few reasons. Your own evaluation of whether I have the right to consider your response rude is certainly not one of them. In fact, helping you with this problem has nothing to do with you at all. I like solving problems, and some other visitor to the site with a similar problem might find this thread later. So, if you still want some help, perhaps you could take what's been offered and ask for more if that truly doesn't help. Otherwise, there are three people who have already given our time to give you tips you have openly and curtly refused to consider. Troubleshooting is an iterative process. Until you rule out those tips as unhelpful through actual thought and reasoning and not a knee-jerk general display of self-importance and infallibility, there's little reason for us to devote any more time to you or your problem.

My one very specific tip to give you right now about your code rather than your behavior is this: Be sure the size of @hosts is smaller than your ulimit for file descriptors and see if the problem goes away. Keep another array outside the sub and only feed the sub ulimit minus some amount (at least five or so to account for the STDIN, STDOUT, STDERR, and a couple of other open files) to the sub at a time. If that fixes your problem, BTW, then my original diagnosis which you completely ignored was the exact right one.

I've discovered some interesting new facts. I added in a ton of debug code and the handles strictly speaking are being closed. However there are *other* handles showing up and subsequently not going away. I suspect they are the STDIN and STDERR of the processes hanging around.

The new code:

sub mping {
    my $mypid = $$;
    print "debug mping\nmy pid is $mypid\n";
    {my @ls = `ls -1 /proc/$mypid/fd`; chomp @ls;
    printf "%s file descriptors:\n", scalar @ls;
    print join " ", sort {$a <=> $b} @ls; print "\n\n"; }
    my @hosts = @{$_[0]};
    @hosts or die "no hosts to mping";
    my ($pingcmd, $pingsize, $pingnumber, $pingtimeout) = @_[1,2,3,4];
    croak "invalid mping arguments" unless $pingcmd and $pingnumber an
+d $pingsize;
    my @pings;    my %results;

    for ($i=0;$i<=$#hosts;$i++) {
        $pings[$i][0] = $hosts[$i];
        my $pid;
        $pid = open $pings[$i][1], "-|", "$pingcmd $hosts[$i] $pingsiz
+e $pingnumber" or 
            die "Error executing $pingcmd: $!\n";;
        my $old_flags = fcntl($pings[$i][1], F_GETFL, 0) 
            or die "can't get flags: $!";
        fcntl($pings[$i][1], F_SETFL, $old_flags | O_NONBLOCK) 
            or die "can't set non blocking: $!";
        $pings[$i][2] = $pid;
        printf "spawned ping, pid $pings[$i][2], fileno %s\n", fileno 
+$pings[$i][1];
        print "$pingcmd, $hosts[$i], $pingsize, $pingnumber failure\n"
+ unless $pid;
    }
    printf "spawned %s ping commands\n", scalar @pings;
    my $start = time; my $j = 1;

    READ: while (@pings) {
        print "starting read loop $j\n";
        printf "there are %s pings active\n", scalar @pings;
        print "fd status from /proc/$mypid/fd:\n";
        {my @ls = `ls -1 /proc/$mypid/fd`; chomp @ls;
        print join " ", sort {$a <=> $b} @ls; print "\n"; }
        for ($i=$#pings;$i>=0;$i--) {
            my $buf;
            my $len = sysread $pings[$i][1], $buf, 580;
            if (not defined $len) { # loop
            }
            elsif ($len > 0) { # read, loop
            }
            elsif ($len == 0) {
                printf "done reading from pid $pings[$i][2] fd `%s`, c
+losing\n", fileno $pings[$i][1];
                my $rv;
                $rv = close $pings[$i][1];
                printf "close $pings[$i][2] status: fd: `%s`; rv: `$rv
+`; \$!: `$!`; \$?: $?\n", 
                    fileno $pings[$i][2];
                $results{$pings[$i][0]}{status} = $? >> 8;
                splice @pings, $i, 1;
            }
            else { die; }
        }
        sleep 2;
        print "end read loop $j\n";
        printf "there are %s pings active\n", scalar @pings;
        print "fd status from /proc/$mypid/fd:\n";
        {my @ls = `ls -1 /proc/$mypid/fd`; chomp @ls;
        print join " ", sort {$a <=> $b} @ls; print "\n"; }
        $j++;
    }
    return \%results;
}
[download]

And the output:

pinging 6 IPs...
debug mping
my pid is 13884
7 file descriptors:
0 1 2 3 4 5 6

spawned ping, pid 13901, fileno 6
spawned ping, pid 13902, fileno 9
spawned ping, pid 13903, fileno 12
spawned ping, pid 13904, fileno 15
spawned ping, pid 13905, fileno 18
spawned ping, pid 13906, fileno 21
spawned 6 ping commands
starting read loop 1
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
end read loop 1
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
starting read loop 2
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
end read loop 2
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
starting read loop 3
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
end read loop 3
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
starting read loop 4
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
done reading from pid 13906 fd `21`, closing
close 13906 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13905 fd `18`, closing
close 13905 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13904 fd `15`, closing
close 13904 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13903 fd `12`, closing
close 13903 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13902 fd `9`, closing
close 13902 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13901 fd `6`, closing
close 13901 status: fd: ``; rv: `1`; $!: ``; $?: 0
end read loop 4
there are 0 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 10 11 13 14 16 17 19 20 22 23

pinging 6 IPs...
debug mping
my pid is 13884
19 file descriptors:
0 1 2 3 4 5 6 7 8 10 11 13 14 16 17 19 20 22 23

spawned ping, pid 13920, fileno 6
spawned ping, pid 13921, fileno 15
spawned ping, pid 13922, fileno 24
spawned ping, pid 13923, fileno 27
spawned ping, pid 13924, fileno 30
spawned ping, pid 13925, fileno 33
spawned 6 ping commands
starting read loop 1
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36
end read loop 1
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36
starting read loop 2
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36
end read loop 2
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36
starting read loop 3
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36
end read loop 3
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36
starting read loop 4
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36
done reading from pid 13925 fd `33`, closing
close 13925 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13924 fd `30`, closing
close 13924 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13923 fd `27`, closing
close 13923 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13922 fd `24`, closing
close 13922 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13921 fd `15`, closing
close 13921 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13920 fd `6`, closing
close 13920 status: fd: ``; rv: `1`; $!: ``; $?: 0
end read loop 4
there are 0 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 16 17 18 19 20 21 22 23 25 26 28 29
+ 31 32 34
35

pinging 6 IPs...
debug mping
my pid is 13884
31 file descriptors:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 16 17 18 19 20 21 22 23 25 26 28 29
+ 31 32 34
35

spawned ping, pid 13936, fileno 6
spawned ping, pid 13937, fileno 27
spawned ping, pid 13938, fileno 36
spawned ping, pid 13939, fileno 39
spawned ping, pid 13940, fileno 42
spawned ping, pid 13941, fileno 45
spawned 6 ping commands
starting read loop 1
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
end read loop 1
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
starting read loop 2
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
end read loop 2
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
starting read loop 3
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
end read loop 3
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
starting read loop 4
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
done reading from pid 13941 fd `45`, closing
close 13941 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13940 fd `42`, closing
close 13940 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13939 fd `39`, closing
close 13939 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13938 fd `36`, closing
close 13938 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13937 fd `27`, closing
close 13937 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13936 fd `6`, closing
close 13936 status: fd: ``; rv: `1`; $!: ``; $?: 0
end read loop 4
there are 0 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 28 29 30
31 32 33 34 35 37 38 40 41 43 44 46 47

pinging 6 IPs...
debug mping
my pid is 13884
43 file descriptors:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 28 29 30
31 32 33 34 35 37 38 40 41 43 44 46 47

spawned ping, pid 13952, fileno 6
spawned ping, pid 13953, fileno 39
spawned ping, pid 13954, fileno 48
spawned ping, pid 13955, fileno 51
spawned ping, pid 13956, fileno 54
spawned ping, pid 13957, fileno 57
spawned 6 ping commands
starting read loop 1
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 5
+3 54 55 56
 57 58 59 60
end read loop 1
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 5
+3 54 55 56
 57 58 59 60
starting read loop 2
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 5
+3 54 55 56
 57 58 59 60
end read loop 2
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 5
+3 54 55 56
 57 58 59 60
starting read loop 3
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 5
+3 54 55 56
 57 58 59 60
end read loop 3
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 5
+3 54 55 56
 57 58 59 60
starting read loop 4
there are 6 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 5
+3 54 55 56
 57 58 59 60
done reading from pid 13957 fd `57`, closing
close 13957 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13956 fd `54`, closing
close 13956 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13955 fd `51`, closing
close 13955 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13954 fd `48`, closing
close 13954 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13953 fd `39`, closing
close 13953 status: fd: ``; rv: `1`; $!: ``; $?: 0
done reading from pid 13952 fd `6`, closing
close 13952 status: fd: ``; rv: `1`; $!: ``; $?: 0
end read loop 4
there are 0 pings active
fd status from /proc/13884/fd:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36 37 38 40 41 42 43 44 45 46 47 49 50 52 53 55 56 5
+8 59

pinging 6 IPs...
debug mping
my pid is 13884
55 file descriptors:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
+ 27 28 29
30 31 32 33 34 35 36 37 38 40 41 42 43 44 45 46 47 49 50 52 53 55 56 5
+8 59

.....etc.....
[download]

So as u can see, the fd of the handle I open does indeed disappear from the /proc output. However on each call to the subroutine, the total count goes up by 12. I ping 6 IP's each time. That leads me to believe STDIN and STDERR are sticking around.

I'm totally mystified by ur earlier umbrage but I appreciate ur consideration.