lpoht has asked for the wisdom of the Perl Monks concerning the following question:

I've been working on a CGI script which completes the following tasks:
1. Get an ID from the query string
2. Gets a filename associated with that ID using Win32::ODBC
3. Loads that filename, which happens to be a jpeg file, into a variable $image
4. Scales, crops, manipulates the image a bunch of different ways
5. Outputs it to the browser

I've been coding it on a 800Mhz test server, and have been generally displeased with the speed of the computations. However, knowing that I would eventually run it on a 2.4Ghz server, I felt reassured that it would run faster. I expected the sript to run 2-3 times faster than it originally had, since the processor in the new machine is 3 times as quick (and the other hardware is faster, too, see below). However, I installed the script on the quick machine, and found that I don't even get 2x speeds. What gives!?

I wrote some timing mechanisms to test individual parts of the script, and found that some of the non-ImageMagick parts hadn't sped up much. These include the DB call and parsing the querystring. Fine, these sections take only 0.05 seconds to execute, and are limited by more than code and hardware (meaning, they rely on more than just ImageMagick, my code, and the machine's hardware to run). Looking at the image manipulation code, though, shows an increase in speed, but again not even 2x. Beacuse these are large images (i.e. 3200x1200), I don't expect a run time of .5 seconds, but the same script that ran in 6 seconds on the old machine executes in a mildly superior 3.5 seconds.

I'm wondering what, in general, might keep this code from running even 2 times faster on a machine which is at least 3 times "better" than another. Below are some basic specs for the two machines:

Old:
PIII 800Mhz
512MB PC133 Ram
ATA100 7200RPM disks

New:
P4 2.4Ghz
512MB PC2100 DDR Ram
ATA133 7200RPM disks
They are both running Win2k server with the latest versions of ActiveState's perl and ImageMagick. Any ideas? Thanks

Replies are listed 'Best First'.
Re: ImageMagick too slow
by sauoq (Abbot) on Dec 27, 2002 at 01:04 UTC

    You shouldn't expect performance to increase linearly with the speed of the processor. Generally, it doesn't. You should also consider other factors. The bottleneck may be elsewhere, such as disk and/or RAM. There may be more load on one machine. Etc. My gut feeling is that a drop from 6 seconds to 3.5 seconds is actually pretty good given the differences you cite.

    -sauoq
    "My two cents aren't worth a dime.";
    
Re: ImageMagick too slow
by John M. Dlugosz (Monsignor) on Dec 27, 2002 at 02:41 UTC
    Well, the disk speed is only slightly faster on the new machine.

    You could try running the NT Performance Monitor on the Image Magick process to see what the deal is.

    I take it the image manipulation is different every time, so you can't cache the resulting scaled/manipulated image?

    What kind of transformations are you doing? It's probably possible to "stream" the JPEG decoding and processing through a pipeline in one pass, as opposed to reading the whole thing and consuming 12 meg of ram, then performing different passes for each part of the process, then encoding the result.

    By having more specialized code, or something that architected differently then Image Magick, you can keep more in the CPU's faster memory cache and work on the number crunching while waiting on disk IO at the same time.

    Depending on the nature of the transforms, you might reample down and almost certainly crop first, to reduce the work of the image processing.

    —John

      I take it the image manipulation is different every time, so you can't cache the resulting scaled/manipulated image?
      Yes, the number of possible combinations is so high that creating cached copies of each possible image would require too much space and processing time.

      What kind of transformations are you doing? It's probably possible to "stream" the JPEG decoding and processing through a pipeline in one pass, as opposed to reading the whole thing and consuming 12 meg of ram, then performing different passes for each part of the process, then encoding the result.
      ImageMagick currently processes the following transformations: crop, scale, extract channel, composite image, clone, and add decorative frames. I like the idea of computing these transformations as the JPEG is processed. Logically, I think that the transformations don't need a full image to be computed. However, because sometimes five or more of these transformations are computed in a row, and sometimes ImageMagick splits one image into two separate images and then computes transformations, they will need to be loaded into RAM at some point. I sense that your idea of streaming would speed things up quite a bit. I will look futher into this idea - does anyone have any input on this one?

      Thanks for your ideas.

        netpbm might do what you wish. there are hundreds of programs, most suitable for streaming together.

        $ djpeg foo.jpg | pnmscale -xy 100 100 | cjpeg -smoo 10 -qual 50 >smal +l.jpg SEE ALSO anytopnm(1), asciitopgm(1), atktopbm(1), bioradtopgm(1), bmptoppm(1), brushtopbm(1), cmuwmtopbm(1), fitstopnm(1), fstopgm(1), g3topbm(1), gemtopbm(1), giftopnm(1), gould- toppm(1), hipstopgm(1), hpcdtoppm(1), icontopbm(1), ilbmtoppm(1), imgtoppm(1), lispmtopgm(1), macptopbm(1), mgrtopbm(1), mtvtoppm(1), pbmclean(1), pbmlife(1), pbmmake(1), pbmmask(1), pbmpscale(1), pbmreduce(1), pbmtext(1), pbmto10x(1), pbmto4425(1), pbmtoascii(1), pbmtoatk(1), pbmtobbnbg(1), pbmtocmuwm(1), pbmtoepsi(1), pbmtoepson(1), pbmtog3(1), pbmtogem(1), pbmtogo(1), pbmtoi- con(1), pbmtolj(1), pbmtoln03(1), pbmtolps(1), pbmtomacp(1), pbmtomgr(1), pbmtopgm(1), pbmtopi3(1), pbmtopk(1), pbmto- plot(1), pbmtoptx(1), pbmtox10bm(1), pbmtoxbm(1), pbmtoybm(1), pbmtozinc(1), pbmupc(1), pcxtoppm(1), pgmbent- ley(1), pgmcrater(1), pgmedge(1), pgmenhance(1), pgmhist(1), pgmkernel(1), pgmnoise(1), pgmnorm(1), pgmoil(1), pgmramp(1), pgmtexture(1), pgmtofs(1), pgmtolispm(1), pgmtopbm(1), pgmtoppm(1), pi1toppm(1), pi3topbm(1), picttoppm(1), pjtoppm(1), pktopbm(1), pnmalias(1), pnmar- ith(1), pnmcat(1), pnmcomp(1), pnmconvol(1), pnmcrop(1), pnmcut(1), pnmdepth(1), pnmenlarge(1), pnmfile(1), pnmflip(1), pnmgamma(1), pnmhistmap(1), pnmindex(1), pnmin- vert(1), pnmmargin(1), pnmnlfilt(1), pnmnoraw(1), pnmpad(1), pnmpaste(1), pnmrotate(1), pnmscale(1), pnmshear(1), pnmsmooth(1), pnmtile(1), pnmtoddif(1), pnmtofits(1), pnmtops(1), pnmtorast(1), pnmtosgi(1), pnmtosir(1), pnmto- tiff(1), pnmtoxwd(1), ppm3d(1), ppmbrighten(1), ppmchange(1), ppmdim(1), ppmdist(1), ppmdither(1), ppmflash(1), ppmforge(1), ppmhist(1), ppmmake(1), ppmmix(1), ppmnorm(1), ppmntsc(1), ppmpat(1), ppmquant(1), ppmquan- tall(1), ppmqvga(1), ppmrelief(1), ppmshift(1), ppmspread(1), ppmtoacad(1), ppmtobmp(1), ppmtogif(1), ppmtoicr(1), ppmtoilbm(1), ppmtomap(1), ppmtomitsu(1), ppmtopcx(1), ppmtopgm(1), ppmtopi1(1), ppmtopict(1), ppmtopj(1), ppmtopjxl(1), ppmtopuzz(1), ppmtorgb3(1), ppmto- sixel(1), ppmtotga(1), ppmtouil(1), ppmtoxpm(1), ppmtoyuv(1), ppmtoyuvsplit(1), psidtopgm(1), pstopnm(1), qrttoppm(1), rasttopnm(1), rawtopgm(1), rawtoppm(1), rgb3toppm(1), sgitopnm(1), sirtopnm(1), sldtoppm(1), spctoppm(1), spottopgm(1), sputoppm(1), tgatoppm(1), tifftopnm(1), xbmtopbm(1), ximtoppm(1), xpmtoppm(1), xvmini- toppm(1), xwdtopnm(1), ybmtopbm(1), yuvsplittoppm(1), yuvtoppm(1), zeisstopnm(1)
        Perhaps finding or making a tool that will crop and scale in one pass while "loading" will give you most of the benifit. Then you have data for a screen-sized image, which is much smaller than the original. I beleive you need to keep three rows on hand in order to do bicubic interpolation to resample the image. For cropping, you can throw away whole blocks and not even process them; though you still have to read through it because you don't know the length ahead of time.

        Look at the code ImageMagick uses to read a JPEG file. I beleive that is the independent free JPEG source.

Re: ImageMagick too slow
by Beatnik (Parson) on Dec 27, 2002 at 00:57 UTC
    Well ImageMagick is just bloated... there is no real way around it. There are a few alternatives on CPAN... Imager, GD or you can always write your own module (preferably based on some C code :) ).

    Greetz
    Beatnik
    ... Quidquid perl dictum sit, altum viditur.
      Just wanted to give feedback on imager. I just installed Imager 0.41 and replaced my ImageMagick code with imager code and what once took 6 seconds to compute with ImageMagick took 8.5 seconds with Imager. I'm going to persue using lower level tools such as cjpeg, djpeg, etc.
Re: ImageMagick too slow
by {NULE} (Hermit) on Dec 27, 2002 at 01:10 UTC
    I have to agree with sauoq here - the limiting factor is probably your memory speed here since your image is *way* to big to fit in CPU cache. PC2100 DDR operates at 133 Mhz, but you get to use the leading and trailing edge of the clock. That should be twice as fast as PC133 RAM, but usually isn't quite because of slight inefficiencies in the archetecture.

    {NULE}
    --
    http://www.nule.org

Re: ImageMagick too slow
by Willard B. Trophy (Hermit) on Dec 27, 2002 at 03:33 UTC
    If you can live with limitations, piping your file through IJG's cjpeg, jpegtran and djpeg tools will be much faster than ImageMagick.

    --
    $,="\n";foreach(split('',"\3\3\3c>\0>c\177cc\0~c~``\0cc\177cc")) {$a++;$_=unpack('B8',$_);tr,01,\40#,;$b[$a%6].=$_};print@b,"\n"

Re: ImageMagick too slow
by fglock (Vicar) on Dec 27, 2002 at 14:25 UTC

    I think ImageMagick for Windows is more reliable than djpeg/cjpeg for Windows. I wouldn't recommend the change.

    ImageMagick uses newer jpeg routines, while djpeg gives me "out of memory" sometimes (although I have more than enough memory). I didn't notice enough difference in processing speeds between ImageMagick and djpeg.

      It only uses newer routines for the completely different JPEG 2000 format. It uses the IJG code for its bog-standard JFIF JPEG file handling.

      --
      $,="\n";foreach(split('',"\3\3\3c>\0>c\177cc\0~c~``\0cc\177cc")) {$a++;$_=unpack('B8',$_);tr,01,\40#,;$b[$a%6].=$_};print@b,"\n"