I believe you can only rely on $! being set after you've already detected an error. Otherwise it can contain garbage. In this case, you're dying because there's some garbage in there:

        if ($!) { 
            print STDERR "ERROR $!\n";
        }
[download]

I didn't see it in the documentation, but on my system $nf is set to -1 in the event of an error from select. Checking this instead of $! might fix your problem.