Personally I would prefer to use one of the standard Perl web frameworks (Dancer, Mojolicious or Catalyst). Dancer is the most minimal, but all 3 have a fair number of dependencies. Your requirements may be simple now, but these things have a way of growing, and you may then find that your lightweight choice will suffer growing pains.

One low-level option, if you really insist on it, is HTTP::Daemon. That's a subclass of IO::Socket::IP, which means you can play with sockets directly if you choose to. If you don't want to use the module directly, its source code might give you some inspiration.