One problem is where and when the _Inline build directory gets installed. If you don't plan ahead, and you're basing a module on Inline::C, the default behavior isn't usually optimal.

There is a proof of concept one good way to do this if you look at Math::Prime::FastSieve. (It used to be the easy way until recently, but the method demonstrated in that module has probably been made obsolete.)

The new easy way uses Inline::Module to assist in creating a distribution that, once installed on the end user's system, looks and feels light-weight, like a plain old XS module. The Inline::Module::Tutorial has pretty good examples that should help you get through the next step.