Do I HAVE to Fork or thread etc.?

No, you can use a select loop. If you're not already familiar with the unix select system call, Perl's builtin might be a little intimidating.

In that case, check out IO::Select. It puts a friendly object-oriented wrapper around the standard call. Also, note that there's an example of a socket server in IO::Select's docs. This should be all you need to get started.