in reply to help with XS pointers

I'm no XS expert, but I have gone through these pains before. Can you share your entire source tree? If you're not comfortable doing so, you might want to look at something relatively straightforward, like Image::Similar (not my module):

In Section Seekers of Perl Wisdom