Whats the best way to read in a very large amount of data from an external program
Pipes! See for example perlopentut for some piping examples.
I think not and AFAIK the only way to do this would be to modify the nfdump command to pause execution when blocked by my program (or use files but I prefer not to really). I wonder if its possible to block execution of the external program without modifying it.
Writing to a pipe whose buffer size is exceeded does in fact block, so unless the program that writes stuff takes special care to do non-blocking writes, pipes already do what you want.
Isn't that wonderful?