You can open() a program to write to it, or open() a program to read from it, but to do both you'll need to take extra steps. You're trying to do both, so you'll need to look for alternatives. One possibility (untested) is to try something like:

open(ARGS, ">args$$") or die "args$$: $!";
print ARGS <<ENDARGS;
new
size 256,256
... etc
ENDARGS
close(ARGS);

open(FLY, "$flyprog -f args$$ |") or die "$flyprog: $!";
binmode(STDOUT);
print "Content-type: image/png\n\n";
print <FLY>;
close(FLY);
unlink("args$$");
[download]

In this approach, you write the commands to fly out to a temporary file, then feed that file back to fly via the "command line" that you're opening. (I'm assuming that there's way to pass a filename to fly.)

The output, which you read from the command, is the PNG.

Depending on your server configuration, you might need to put the temporary "args" file somewhere else, like /tmp