in reply to Re^2: IO::Uncompress::Gunzip thread safe?
in thread IO::Uncompress::Gunzip thread safe?

If you can make use of multiple CPUs, it might be easier to handle the decompression through an external process, at the cost of more inter-proces IO:

open my $fh, "gzip -cd $file |"
    or die "Couldn't read from '$file': $! / $?";
binmode $fh;

while (<$fh>) { # or whatever loop mechanism is appropriate
    ...
}

[download]

That way you lose some finer grained control over the error states - for zero-byte files, gzip might just exit and not output anything and your program might think everything is OK, for example.

Replies are listed 'Best First'.
Re^4: IO::Uncompress::Gunzip thread safe?
by chris212 (Scribe) on Nov 21, 2016 at 21:55 UTC
    That works pretty well on Linux. It won't work on Windows, but better than nothing. Thanks!
[reply]
      It won't work on Windows,

      Why not? 

      C:\test>dir *.gz
 Volume in drive C is Local Disk
 Volume Serial Number is 8C78-4B42

 Directory of C:\test

30/12/2014  13:59           228,720 01mailrc.txt.gz
20/03/2009  15:58        73,773,666 chr1.fa.gz
20/03/2009  16:02        11,549,785 chr21.fa.gz
20/03/2009  15:59        54,997,756 chr6.fa.gz
21/11/2016  19:03            13,203 test.gz
               5 File(s)    140,563,130 bytes
               0 Dir(s)  366,120,411,136 bytes free

C:\test>p1
[0]{} Perl> open GZ, 'gunzip -c test.gz |' or die $!; print while <GZ>
+;;
1 qwertyuiopasdfghjklzxcvbnm

2 qwertyuiopasdfghjklzxcvbnm

3 qwertyuiopasdfghjklzxcvbnm

4 qwertyuiopasdfghjklzxcvbnm

5 qwertyuiopasdfghjklzxcvbnm

6 qwertyuiopasdfghjklzxcvbnm

7 qwertyuiopasdfghjklzxcvbnm

8 qwertyuiopasdfghjklzxcvbnm

9 qwertyuiopasdfghjklzxcvbnm
...

      [download]
      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]

        I suppose it could work if we had the gzip command on our Windows servers. What provides the gzip command on Windows for you? Security restricts what we can install, but it is worth looking into.

        Also, I noticed that providing the command arguments to open in as a list rather than a single string doesn't work in Windows? Any suggestions for preventing unusual file name characters from causing problems on Linux and Windows?

[reply]

      I've used this approach on both, Windows and unixish OSes with great success. Where did it fail for you and how?

[reply]

In Section Seekers of Perl Wisdom