anaras has asked for the wisdom of the Perl Monks concerning the following question:

Hi there, This is the first time I am posting a message. I need to compress (gzip) and make chunked output for a multiple content blocks (chunks). First, I need to compress (gzip) and output the 1st content block. Then do some other stuff and compress (gzip) and output the remaining 1 or 2 content blocks. The problem is if I only process and serve the entire content in 1 combined block, it works. But, I need to process the first block, do something else and process the remaining 1 (or 2 content blocks). Below is the code. Obviously I am doing something wrong. Asking for help and advice. Thanks in advance!
#!/usr/bin/perl # ------------------------------------- use strict; use Compress::Zlib; # ------------------------------------- # ------------------------------------- my $content1 = "<p>This is content part1</p>"; my $content2 = "<p>This is content part2</p>"; my $content3 = "<p>This is content part3</p>"; #my $apply_encoding_type = 'deflate'; my $apply_encoding_type = 'gzip'; my $this_line_break = "\r\n"; my $status; # ------------------------------------- # compress and prepare '$content1' # ------------------------------------- my $compressed_content1 = ''; $status = &compress_content($apply_encoding_type,\$content1,\$compress +ed_content1); unless($status eq 'ok'){ print "Content-type: text/html\n\n"; print "<p>Problem-1: $status </p>"; exit; } my $hexadecimal_length_compressed_content1 = sprintf("%X", length($com +pressed_content1)); # ------------------------------------- # output compressed '$content1' chunk ! # ------------------------------------- # First HTTP Response Headers print "Cache-Control: private, max-age=0, must-revalidate",$this_l +ine_break; print "Vary: Accept-Encoding",$this_line_break; print "Content-Encoding: $apply_encoding_type",$this_line_break; print "Transfer-Encoding: chunked",$this_line_break; print "Content-type: text/html; charset=UTF-8",$this_line_break,$t +his_line_break; # Now, output compressed '$content_part1' print $hexadecimal_length_compressed_content1,$this_line_break; print $compressed_content1,$this_line_break; # ... do something else ... # ... do something else ... # ... do something else ... # ... do something else ... # ... do something else ... # ... do something else ... # ------------------------------------- # compress and prepare '$content2' # ------------------------------------- my $compressed_content2 = ''; $status = &compress_content($apply_encoding_type,\$content2,\$compress +ed_content2); unless($status eq 'ok'){ print "Content-type: text/html\n\n"; print "<p>Problem-2: $status </p>"; exit; } my $hexadecimal_length_compressed_content2 = sprintf("%X", length($com +pressed_content2)); # ------------------------------------- # output compressed '$content2' chunk ! # ------------------------------------- print $hexadecimal_length_compressed_content2,$this_line_break; print $compressed_content2,$this_line_break; # . # . # . # Now, do the same for the final '$content3' # ------------------------------------- # compress and prepare '$content3' # ------------------------------------- my $compressed_content3 = ''; $status = &compress_content($apply_encoding_type,\$content3,\$compress +ed_content3); unless($status eq 'ok'){ print "Content-type: text/html\n\n"; print "<p>Problem-3: $status </p>"; exit; } my $hexadecimal_length_compressed_content3 = sprintf("%X", length($com +pressed_content3)); # ------------------------------------- # output compressed '$content3' chunk ! # ------------------------------------- print $hexadecimal_length_compressed_content3,$this_line_break; print $compressed_content3,$this_line_break; # ------------------------------------------------- # Now, finish with the final (zero length) header/s # ------------------------------------------------- print "0", $this_line_break, $this_line_break; exit; ########################################### # END OF MAIN SCRIPT ! ########################################### sub compress_content{ #my($apply_encoding_type,$content_ref,$compressed_content_ref,$end_of_ +stream) = @_; my($apply_encoding_type,$content_ref,$compressed_content_ref) = @_; $apply_encoding_type = $apply_encoding_type || 'gzip'; my($d, $out1, $out2, $status); # ------------------------------------- if($apply_encoding_type eq 'gzip'){ ($d,$status) = deflateInit( -WindowBits => 16 + &MAX_WBITS() ); } else{ # ------------ # 'deflate' ? # ------------ ($d,$status) = deflateInit(); } # ------------------------------------- # ------------------------------------- unless(defined($d)){ my $err_msg = $status; print "Content-type: text/html\n\n"; print "<P>DEFLATE-INIT ERROR: $err_msg (\$apply_encoding_type: $ap +ply_encoding_type)<P>"; exit; #return "fail-1 ($err_msg); } # ------------------------------------- # ------------------------------------- ($out1, $status) = $d->deflate($content_ref); unless($status == Z_OK){ my $err_msg = $d->msg(); #my $err_msg = $status; print "Content-type: text/html\n\n"; print "<P>COMPRESS-ERROR: $err_msg (\$apply_encoding_type: $apply_ +encoding_type)<P>"; exit; #return "fail-2 ($err_msg); } # ------------------------------------- # ------------------------------------- if($apply_encoding_type eq 'gzip'){ #if($end_of_stream){ # ($out2, $status) = $d->flush(Z_FINISH); #} #else{ # ($out2, $status) = $d->flush(Z_SYNC_FLUSH); #} ($out2, $status) = $d->flush(Z_SYNC_FLUSH); } else{ # ------------ # 'deflate' ? # ------------ ($out2, $status) = $d->flush(Z_SYNC_FLUSH); } # ------------------------------------- # ------------------------------------- unless($status == Z_OK){ my $err_msg = $d->msg(); #my $err_msg = $status; print "Content-type: text/html\n\n"; print "<P>DEFLATE-FINISH ERROR: $err_msg (\$apply_encoding_type: $ +apply_encoding_type)<P>"; exit; #return "fail-3 (error: $err_msg, status: $status)"; } # ------------------------------------- # Finally, add 'flush' output data to the initially deflated data '$ou +t1') !!! if(defined($out2)){ $out1 .= $out2; } # ------------------------------------- $$compressed_content_ref = $out1; return 'ok'; } ############################################ # END OF EVERYTHING !!! ############################################

Replies are listed 'Best First'.
Re: zlib compression question
by talexb (Chancellor) on Aug 31, 2016 at 15:22 UTC

    Having only read through your code once, I'm not clear as to why you're trying to compress stuff in chunks. You're also doing this as a web page, which complicates things.

    My suggestion would be to get the compression code figured out from the command line, and once that's working properly, use that code to build a web page.

    From a code smell point of view, you don't need 1; at the end of a script -- that's only required for modules, since they are used by scripts, and that usage wants to return TRUE, aka 1.

    For a posting to this site, I'd also recommend deleting any development code that's commented out .. it's useful to you, but isn't pertinent. Ideally, a code posting is a slimmed down, as-simple-as-possible example that shows the specific problem.

    Alex / talexb / Toronto

    Thanks PJ. We owe you so much. Groklaw -- RIP -- 2003 to 2013.

Re: zlib compression question
by FreeBeerReekingMonk (Deacon) on Sep 02, 2016 at 15:50 UTC
    DELETED (proposed solution did not work)

    It involved the fact that you compress 3 times, with 3 headers instead of one single header. But zlib not only compresses data, it encapsulates the raw DEFLATE data by adding a header and trailer. This means that when you unchunk it, you have 1 data blob/file, with 3 headers/trailers inside it. So I wondered if that was being decompressed properly.

    See Chunked_transfer_encoding