Not that I expect it accounts for all of the speed differences, but Imager draws anti-aliased filled polygons, while GD doesn't.

Pre-calculating the 'white' with something like:

my $white = Imager::Color->new("white");
...
$img->polygon (points=>\@polygon, color=>$white);
[download]

may also help a little, though it's likely to drown in the worked needed to actually draw the polygons.

Imager has other inefficiencies in filling areas that probably make some difference too.