Thank you for your elaborate answer and demonstration of the relevant code snippet. Unfortunately, I cannot reproduce your problem because for me it works completely well and does not run out of memory, neither when using my snippet nor when using grep. I guess that this must be due to differences in our code.

We could now play a fun round of "Guess My Code", where I guess a letter of your program and you tell me whether the actual letter is higher or lower. But then again, you could invest some effort into showing a relevant part of your code yourself. You should also read XY Problem, because I have the distinct feeling that you're trying to solve a problem that you wouldn't have when using a better approach.

If you want to keep hanging onto your idea, you should realize that instead of system, where you have no control over the spawned process (as documented), open returns the pid of the child and kill can be used to terminate the child process. But then, why one would do that when there likely are far better solutions without spawning child processes escapes me.