"The result seems to be that Perl keeps gobbling memory as long as the process generates output, then when the process ends it returns the appropriate slice. "

That it does. Perl doesn't know what the command might do, so it can't take a slice until it returns. And cat reads everything (this is why it's also a bad idea to use something like 'cat file.txt | more' instead of 'less file.txt).

Also, it's generally not such a good idea to shell out to an external command when you have that same functionality built it.