in reply to On optimizing nested loops

That's Interesting. Does using grep show any significant improvement over just swapping the order of the foreach loops in your first filter?

I wonder if using each will have any impact? I'll have to give it a go and see.

So maybe something like this?

 while (my ($field,$value) = each %{$where}) {
      @in = grep { $_->{$field} eq $value } @in;
 }

[download]

Replies are listed 'Best First'.
Re^2: On optimizing nested loops
by FloydATC (Deacon) on Oct 20, 2014 at 06:58 UTC
    I could not measure any difference between using foreach/keys and while/each.

    I'm thinking maybe there's a way to turn my $where hashref into a single expression usable by grep() without making it work significantly slower.

    -- FloydATC

    Time flies when you don't know what you're doing

[reply]

      If @in is fairly long, then I'd certainly expect this:

      @in = grep { $_->{$field} eq $value } @in;

      [download]

      To be measurably faster than:

      @out = grep { $_->{$field} eq $value } @in;
@in = @out;

      [download]

      Another optimization you could try is to note that each field you process cuts down the size of @in, so it makes sense to first process the fields that will cut it down the most. Filtering on gender eq "f" is probably going to cut out around 50% of @in, but filtering on surname eq "Davis" is likely to cut out around 99%.

      Right now, you're iterating through keys %$where in no particular order - i.e. hash order. Sorting keys %$where to prioritize keys where the data exhibits a high degree of variation ought to speed many searches up significantly, and the only cost will be sorting what is presumably a fairly small list of hash keys.

      toby döt ink
[reply]
[d/l]
[select]

        Good point.

        For my particular application I rarely get more than one or two $where keys, and most of them match less than 1% of the full data set, but for other uses I'm sure it would pay off to use a reference to an array of key/value pairs so the match order could be optimized.

        -- FloydATC

        Time flies when you don't know what you're doing

[reply]

In Section Meditations