As a matter of fact, as your original question is "how much time does it take for my script to get the image", I'd believe what ab says. Then we have to understand why it takes 3.5 seconds, but as no caching is being done it is the actual time it takes to serve your request (by the way, how many concurrent sessions do you use on ab and how many hits do you make? the number of concurrent sessions can place a good strain on performance).
From your profile, it seems that everything you do is comparatively small after you pay 2.1 seconds of startup time. I'd try mod_perl first, then FastCGI, and then maybe I'd play with YAML. The rest of the code is not worth optimizing at this stage. And use ab to benchmark it all, so you have consistent and actual execution results within the web server.
