I think one of the biggest challenges will be handling the C++ templates (like the vector in the sample you've shown).

Inline::CPP doesn't seem to support them at all, and writing the XS code yourself that can handle them for non-trivial data structures will probably take some time to get familiar with (in particular considering your "disclaimers").

You could try Swig, which claims to support templates — but I must say that when I personally had tried a project with Swig some time ago, my experiences with complex (and in particular dynamic) data structures were not the best, even though I didn't have to deal with templates... (after things had turned into a non-debuggable chaos, I eventually switched to plain XS, and things became a lot easier). That was about a decade ago, though, so I'm not sure in what ways Swig might have improved (I haven't used it since then).  Good luck, anyhow.