You might use profiling to really check where the time is spent. Maybe you do something very inefficient. See Profiling your code.

You might post the code of what you are doing. How can we improve code we don't know?

There is also a mailing list for PDF::API2 users (check the wiki on the sourceforge page of the project). If PDF::API2 is the culprit they might have the inside knowledge you need.