in reply to Re^3: Nested greps w/ Perl
in thread Nested greps w/ Perl

I have a flat (test file) that looks like this:

Tommy Z
Tommy Z
Chris Z
Chris B
Chris Z
Jake Z
Jake Y

I'm simply counting how many Z's each person has and ignoring all other letters.

Output would look like

Tommy 2
Chris 2
Jake 1

Replies are listed 'Best First'.
Re^5: Nested greps w/ Perl
by Laurent_R (Canon) on Dec 20, 2016 at 19:02 UTC
    OK, here is a test. I started with a word.txt file having 113809 unique entries: 
    $ wc words.txt
 113809  113809 1016714 words.txt

    [download]
    From there, I created a test_file.txt file, in which one 1200-character string was added to each entry of the word.txt file, and then copied 26 times, giving me a 3-million-line and 3.5 GB test_file.txt test file: 
    $ perl -ne 'chomp; for my $let (A..Z) { print  "$_ ". "foobar" x 200, 
+" $let\n" }' words.txt > test_file.txt
$ wc test_file.txt
   2959034    8877102 3586152466 test_file.txt

    [download]
    Then, I ran a program counting for each word the number of occurrence of a 'Z': 
    use strict;
use warnings;
use feature qw/say/;

my %histogram;
open my $IN, "<", "test_file.txt" or die "could not open input file $!
+";
while (<$IN>) {
    chomp;
    my ($key, undef, $val) = split / /, $_;
    $histogram{$key}++ if defined $val and $val eq 'Z';
}
print scalar keys %histogram;

    [download]
    Running it takes less than 14 seconds: 
    $ time perl test_Z_bigfile.pl
113809
real    0m13.973s
user    0m11.734s
sys     0m2.140s

    [download]
    My input data file certainly does not look like yours, but I could count all entries ending with a "Z" from a 3-million-line and 3.5 GB test file in 14 seconds (on a simple laptop). I fail to see why it should take centuries on your side.

    So, again, you're probably not telling everything or not doing it the right way.

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom