To support things like caller, Perl (and other languages) use something called call frames. This is an internal data structure that represents a single call into a subroutine. (That's sort of a convenient lie; it's not quite an internal data structure in Perl, but it's a convenient fiction.)

Every time you call a new subroutine, Perl makes a new call frame and executes more code. All of those calls add up.

There is a particular program optimization called tail recursion that a compiler can perform when it notices that the last call into a subroutine is recursive, but Perl 5 doesn't do that. (It's too bad, because it can save lots of memory and time, and it's not difficult to get back the accounting information that caller might need.) There are other optimizations too that allow you to transform recursive calls into iterative calls, but Perl doesn't do those for you either.