Update:
I have removed all the encoding stuff, and the problem remains.
So the cause seems not the encoding, like I thought first.

Maybe it has to do with the fact that the script is a script spawned by the CGI, via double-forking, setsid and exec?

Now I ask myself, could this buffering issue be circumvented by not doing sequential file access, but instead random-access( eg. seek+print)?
I will check that tomorrow and report back.