I don't know of any Windows equivalent of ulimit -f. (ie. per process filesize limits.)

Your limiting tee idea seems the simplest to implement, but be aware that unless the writing process is checking for failing writes, it may not notice the pipe has gone away.

Eg: When the second instance of perl below terminates, the first instance blythly continues to write to the pipe. It doesn't cause any memory growth, but it does use a substantial amount of cpu until it decides to stop writing of it's own accord.

perl -E"say $_ for 1..100e6" | 
perl -E"while(<>){ $n += length; last if $n > 1024**2; warn qq[$n\n];}
+"
[download]

If the first instance was checking for the success of it writes, it would notice the pipe going away:

perl -E"say or die $^E for 1 .. 100e6" |
perl -E"while(<>){ $n += length; last if $n > 1e5; warn qq[$n\n]; }"

...
99990
99996
The pipe is being closed at -e line 1.
[download]

But if you can't change the process that's probably not useful info :(