Then run your server under the debugging perl. When it segfaults and dumps core, open up the core file in gdb (gdb -c core) and do a backtrace (bt) to see where it's crashing.

Until you know where the crash is coming from, you're a bit stuck. (Like others here, I'm inclined to doubt that the problem is with IO::Socket or IO::Select, because those modules are both widely used and well-tested, but it's not impossible.)