in reply to Re: Memory issue with large array comparison
in thread Memory issue with large array comparison

My understanding is that he has two arrays: the first containing 50,000+ pathnames, and the second containing 10,000 strings which may match a filename in one of the pathnames. I don't see how you exclude those matches with a regex, unless you concat all 10,000+ strings together with pipe symbols, and I doubt a regex like that would be very efficient.

Aaron B.
Available for small or large Perl jobs; see my home node.

  • Comment on Re^2: Memory issue with large array comparison

Replies are listed 'Best First'.
Re^3: Memory issue with large array comparison
by ww (Archbishop) on May 25, 2012 at 02:20 UTC

    You may be right. Certainly the regex you envision would be a tad clumsy. :)

    At the moment, it's looking to me as though I should have written the code to support my claim re regex solutions, before posting, rather than simply doing a mental sketch (which proved to be wrong in at least that particular instance). Thank you for making me "think, again!"

    However, on the theory that simple (or, in this case, 'concise') is desirable:

    #!/usr/bin/perl use 5.014; # 972317 my @arr1 = ('C:\abc\def\zz456','C:\abc\xyz\ab123','C:\abb\bba\ac234',' +C:\abc\xyz\ab321'); my @arr2 = qw/ab123 ac234 ad456 ab321/; # no match for C: +\abc\def\zz456 my $dir_entry; my @arr3 = grep { $dir_entry = $_; not grep { $dir_entry =~ /\Q$_/i } +@arr2 } @arr1; say "Following is/are dir entry/entries (in \$arr1) without matching i +tems in \@arr2"; say each(@arr3);

    Output ( index and array element; see each ):
    0C:\abc\def\zz456

    And can some knowledgeable Monk explain why the index of the zeroth element of the array is rendered as 0C?

    Update: A knowledgeable Monk (++) has indeed explained... and the answer has little to do with the assumption immediately above. Meh! Blech on me.

      This is what the original poster did, but with different variable names and a slightly different regex, and it ran out of memory. But isn't this O(N2)? It seems to me that it greps every item in one array against all the items in the other array, so it's really no different from this:

      my @array3; OUTER: for my $a1 (@array1){ for my $a2 (@array2){ next OUTER if they_match_somehow(); } push @array3, $a1; # it didn't match anything in @array2 }

      Both cases have two nested loops; it's just harder to see them in the grep-within-a-grep method.

      Aaron B.
      Available for small or large Perl jobs; see my home node.

        + + aaron_baugher; I didn't even notice the similarity ...and shame on me for that, as it means it's no answer to OP's original dilemma.

        I was, I realize now (thanks to your watchfulness), obsessing on the multiple responses offering use of a hash as a solution. I still think those represent something close to cargo-culting a meme, (rather than actual code) -- but not an optimal solution, since, if I read the wisdom of the sages correctly (and if they're right, of course), using a hash would be at least as memory intensive and probably more so.

        That's also an issue with map and grep (cf Eliya's observations, above), but perhaps less so than using a hash (that's another test that I haven't undertaken, but which might lead to a publishable finding). And in the same node, Eliya makes a cogent point (echoed in slightly different context by dave_the_m's code: there are a variety of ways to attack OP's problem with reduced memory demand. Yet another might be a step-wise solution: first, separate the id portion of the first dataset to a file of it's own; then identify the ids in the second file that don't have identical (or identically normalized, if that's involved, too) values.

        But, again, ++ for casting a sharp eye on the prior responses.

      can some knowledgeable Monk explain why the index of the zeroth element of the array is rendered as 0C?
      Not aspiring at the title of knowledgeable Monk, but the index is 0 and the string starts with C:. You are grepping @arr1 which contains full paths.
Re^3: Memory issue with large array comparison
by CountZero (Bishop) on May 25, 2012 at 13:53 UTC
    I don't see how you exclude those matches with a regex, unless you concat all 10,000+ strings together with pipe symbols, and I doubt a regex like that would be very efficient.
    One solution: Regexp::Assemble

    It creates an optimized regex that checks all your strings at once. It is usually shorter and much faster than your "pipe"d regex. For example, 10,000 strings of each 5 characters were turned into a regex less than 25,000 characters long.

    CountZero

    A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

    My blog: Imperial Deltronics