ulimit -n
[download]

ulimit -n $bigNumber
[download]

ulimit -n 100000 has no effect - it changes the same thing as the setrlimit() call

#!/usr/bin/perl -w
use strict;
use Socket;

open OLDOUT, ">&STDOUT";
for (my $i=0;$i<3;$i++)
{
open STDOUT, ">&", \*OLDOUT or die "reset STDOUT - $!";
my ($in, $out);
my $buf;
open STDOUT, ">/dev/null" or die "redirect - $!";
socketpair( $in, $out, AF_UNIX, SOCK_STREAM, PF_UNSPEC) or die + $!;
printf STDERR "sock is $out file %d\n", fileno ($out);
open STDOUT, ">&=", fileno($out) or die ">&= fileno(fd) - $!";
print "test1\n";
open STDOUT, ">&=", $out or die ">&= fd - $!";
print "test2\n";

open STDOUT, ">&", \*OLDOUT or die "reset2 STDOUT - $!";
}
[download]

Your test works for me on 32 and 64 bit (I originally found the problem using socketpair() :-). The original script also fails for me using perl, v5.8.8 built for i486-linux-gnu-thread-multi. 32bit strace working with 2000 files:

Read more... (4 kB)

and the same process failing with 4000 files:

Read more... (4 kB)

I can replicate the issue with 5.8.x, but not with 5.10.0. The exact limit seems to be 2048 (try something like for my $max (2035..2050) {...} to verify yourself).  A bit of digging in the 5.8.8 sources suggests this might have to do with a constant in perlio.c:

#define PERLIO_MAX_REFCOUNTABLE_FD 2048
[download]

which would make PerlIOUnix_dup() fail if fd >= PERLIO_MAX_REFCOUNTABLE_FD:

PerlIO *
PerlIOUnix_dup(pTHX_ PerlIO *f, PerlIO *o, CLONE_PARAMS *param, int fl
+ags)
{
    PerlIOUnix *os = PerlIOSelf(o, PerlIOUnix);
    int fd = os->fd;
    if (flags & PERLIO_DUP_FD) {
        fd = PerlLIO_dup(fd);
    }
    if (fd >= 0 && fd < PERLIO_MAX_REFCOUNTABLE_FD) {
        f = PerlIOBase_dup(aTHX_ f, o, param, flags);
        if (f) {
            /* If all went well overwrite fd in dup'ed lay with the du
+p()'ed fd */
            PerlIOUnix_setfd(aTHX_ f, fd, os->oflags);
            return f;
        }
    }
    return NULL;
}
[download]

In Perl 5.10.0, the respective number of refcountable file descriptors is dynamically resized as needed...

(In case you really want to know, just modify the value, recompile perl, and try again to test the hypothesis :)

I have found two ways around the limit (with thanks to almut):

1. Run perl with environment variable PERLIO set to :stdio
2. Recompile perl with PERLIO_MAX_REFCOUNTABLE_FD in perlio.c set to a suitably large number

Running perl with PERLIO set to :stdio makes perl use standard IO libraries instead of its own. This avoids the hard coded limit on file descriptor reference counts in perl's IO routines but note the warning in perlio.c:

/* This file contains the functions needed to implement PerlIO, which
 * is Perl's private replacement for the C stdio library. This is used
 * by default unless you compile with -Uuseperlio or run with
 * PERLIO=:stdio (but don't do this unless you know what you're doing)
 */
[download]

I recompiled perl with PERLIO_MAX_REFCOUNTABLE_FD set to 32768 and your test script ran to successful completion. Tests on the recompiled perl ran ok.

Thanks for this solution -- I ran into it & didn't waste much time, thanks to you.

ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff2b62da20) = -1 EINVAL (Invalid argument)

I suspect that the ioctl() you're calling is invalid for the x86_64 architecture. The completely malaprop file descriptor (SNDCTL_TMR_TIMEBASE doesn't have much to do with duping filehandles) certainly implies that this is the case; so does the reported success on the 32-bit machine.

In my tests with perl 5.8.8 on CentOS 5.2, open with ">&=" is failing as soon as or whenever the file descriptor being opened is 2048 or greater. It appears to be failing in perl because strace does not show failed system calls.