Considering how simple your problem is, and the fact that you want maximum speed, have you considered re-writing in C, or another fast compiled language.

In the past when I was working on image processing problems, I often saw a program run take an hour or more in perl, but only take a minute or so when re-coded in C. In my case, the typical problem was to take some bitmap images representing 1080p frames of uncompressed video, and convert them from RGB to YUV or suchlike, and them output them as ASCII text listing pixel values.