Is there a way to specify external C++ files and I'm missing ?

Yes:

use Inline Config =>
    INC => '-I/my/path',
    LIBS => '-L/your/path -lyourlib';

use Inline CPP => <<'END';
/* your C++ code */
END

# your perl code
[download]

See the Inline::CPP documentation - and the Inline::C-Cookbook documentation (as the same principles apply to C++).

Cheers,
Rob