Does anyone know of a more efficient method for finding just the stack depth without using multiple calls to caller() each time you need that number?

I think you'd would have to implement it in XS and adapt the underlying code for caller. Looking at pp_ctl.c in the source, it looks like caller is just walking the stack itself, but someone with more knowledge of XS and Perl guts would have to advise you on just what you need to do.