I would write it using Moose now but the performance hit wouldn't be worth it.

Actually Moose should perform quite well for something like a HTTP server since the Moose performance hit is largely at compile time and if you utilize the proper optimizations in Moose the runtime can be pretty fast (simple Moose accessors are faster then Class::Accessor).