Caveat: I am not at all familiar with the tools you are using.
However, from your brief description, it sounds like you are trying to produce a vignetting effect around a non-rectangular region,using Alpha compositing.
A long time ago in a different life I had a similar problem to solve. I achieved an amazingly convincing result by the following technique. (This is from vague memories going back almost 20 years, so If I miss a step; or re-order them; or use non-current terminology; sorry!):
In reply to Re: goo canvas image mask
by BrowserUk
in thread goo canvas image mask
by Anonymous Monk
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |