Caveat: I am not at all familiar with the tools you are using.

However, from your brief description, it sounds like you are trying to produce a vignetting effect around a non-rectangular region,using Alpha compositing.

A long time ago in a different life I had a similar problem to solve. I achieved an amazingly convincing result by the following technique. (This is from vague memories going back almost 20 years, so If I miss a step; or re-order them; or use non-current terminology; sorry!):

1. Draw a black background.
2. Open a path.
3. Draw your shape(s) with an alpha of 0%.
4. Scale your shape(s) reducing the size by -1%.
5. Redraw the shapes with alpha +10%.
6. Goto step 3; repeat 10 times.
7. Close your path, and stroke it.
8. Use this path as your destination in a Porter-Duff alpha blend XOR operation.

thanks for the reply, but i forgot to mention that the mask Im going to use is of type .png or .jpg. It is easier to create masks on photoshop than creating paths. :D, so i cant use your procedure. but thanks for the reply. :)

This is (of necessity due to GDs limitations) a crude vignetting process. Starting with this, you get this.

#! perl -slw
use strict;
use GD;

sub rgba2n { unpack 'N', pack 'C4', pop, @_ }

my $src = GD::Image->newFromJpeg( $ARGV[0], 1 ) or die $!;

my $dia = $src->width < $src->height ? $src->width : $src->height;
my $maxDia = sqrt( $src->width**2 + $src->height**2 );
my $thickness = 10;

my $alpha = 127;
my $alphaDelta = $thickness * 127 / ( $maxDia - $dia );

$src->setThickness( $thickness );

while ( $dia < $maxDia ) {
    $src->ellipse(
            $src->width / 2, $src->height /2,
            $dia, $dia, rgba2n( 0, 0, 0, $alpha )
    );
    $alpha  -= $alphaDelta;
    $dia += $thickness;
}

open PNG, '>:raw', "vigged.png";
print PNG $src->png;
close PNG;

## Look at the result; starts default image editor.
system 'vigged.png';
[download]

The results are crude because GD doesn't support paths or Porter-Duff operations, but the basic idea of using alpha blending of a shape (here thick-walled circles) over an image with an alpha gradient gives an impression of the result.

In theory, you should be able to construct a mask image the same size as the one being vignetted--generally described as the source). You fill this mask image with black, and then draw successively smaller circles (shapes), also in black but gradually reducing the alpha component to 0. The final circle (shape) being the size of the unblended portion of the source that you wish to show through. You then XOR blend the mask with the original image to produce the required output image.

Which should look something like the second image above, but if done using a properly scaled and stroked path, smoother.

If I understand you correctly--no guarantee--what you are hoping for, is for the black in this example to be replaced by pixels from a second source image. In this case, rather than starting with a mask image filled with black, you construct the mask from the second (overlay) image, by drawing your shape successively smaller with reducing alpha as above. But instead drawing the shape in a given color, you draw it such that it only changes the (embedded) alpha component of the existing pixels. I haven't figured out how to do this with GD--or even if it is possible,

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

RIP an inspiration; A true Folk's Guy

...so i cant use your procedure...

Yes you can, the procedure is exactly the same, except instead of drawing the shape, you import from image

my $pixbuf = Gtk2::Gdk::Pixbuf->new_from_file("girl16.jpg");
    my $i1 = Goo::Canvas::Image->new($canvas->{'item_layer'},$pixbuf,-
+$pixbuf->get_width/2,-$pixbuf->get_height/2);
    $i1->set('operator'=>'over');

    $pixbuf = Gtk2::Gdk::Pixbuf->new_from_file("girl15.jpg");
    my $i2 = Goo::Canvas::Image->new($canvas->{'item_layer'},$pixbuf,-
+$pixbuf->get_width/2,-$pixbuf->get_height/2);
    $i2->set('operator'=>'over');

    $pixbuf = Gtk2::Gdk::Pixbuf->new_from_file("mask.png");
    my $i3 = Goo::Canvas::Image->new($canvas->{'item_layer'},$pixbuf,-
+$pixbuf->get_width/2,-$pixbuf->get_height/2);
    $i3->set('operator'=>'dest_atop');
[download]