I would almost bet that is a CPU problem on Perl's side, specifically caused by CGI.pm. Could you check CPU usage during the test, and CPU burned by checking "times" ?
Comment on Re: mod_perl vs php performance when accessing Oracle under load
I've tried taking out CGI entirely as the current version of the script doesn't use it; I get an average of 1800ms
CPU Usage sits at about 30% during the test. Not sure how I'd go about checking 'times', short of starting apache with 'time' in debug mode - and then I can't do the concurrency tests.