I would try building a debugging perl (i.e. with Configure option -Doptimize=-g), and make sure you can dump core files (check that ulimit -c is large enough, preferably unlimited).

Then run your server under the debugging perl. When it segfaults and dumps core, open up the core file in gdb (gdb -c core) and do a backtrace (bt) to see where it's crashing.

Until you know where the crash is coming from, you're a bit stuck. (Like others here, I'm inclined to doubt that the problem is with IO::Socket or IO::Select, because those modules are both widely used and well-tested, but it's not impossible.)