in reply to Re: Memory utilization and hashes
in thread Memory utilization and hashes

Since I need all of the query and answers info on one line in the output, yes, I need to collect them up until I have all of the answers.

here is a more closely working example of the code. I was trying to keep it simple and focus on the memory usage of the hash but here we are. 

#!/usr/bin/perl

use warnings;
use strict;
$|++;

use JSON;

my $l;
my @vals;
my $json;
my %pairs;
my %pind;
my %flush;

while (<DATA>) {
    $l = $_;
    chomp $l;
    @vals = split /;/, $l;
    if ($vals[0] =~ /Query/) {
        if (! $pairs{$vals[1]}) {
            $pind{$vals[1]} = 0;
        }
        if (!defined $flush{$vals[1]}) {
                $flush{$vals[1]} = " ";
        } elsif ($flush{$vals[1]} ne $vals[1]) {
                $json = encode_json $pairs{$vals[1]};
                print "DEBUG: Flushing \"complete\" answer\n";
                print $json."\n";
                delete $pairs{$vals[1]};
                $flush{$vals[1]} = $vals[1];
                $pind{$vals[1]} = 0;
        }
        $pairs{$vals[1]}{$vals[2]} = $vals[3];
        $pairs{$vals[1]}{id} = $vals[1];
    } elsif ($vals[0] =~ /Answer/) {
        $pairs{$vals[1]}{$vals[0]}[$pind{$vals[1]}++]{$vals[2]} = $val
+s[3];
    }
}

print "DEBUG: output remaining data ...\n";

foreach my $key (keys %pairs) {
        $json = encode_json $pairs{$key};
        print $json."\n";
}

__DATA__
Query;1;host;www.example.com
Answer;1;ip;1.2.3.4
Query;2;host;www.cnn.com
Query;3;host;www.google.com
Answer;2;ip;2.3.4.5
Answer;2;ip;2.3.4.5
Query;4;host;www.google.com
Answer;4;ip;3.4.5.6
Answer;3;ip;3.4.5.6
Query;2;host;www.example2.com
Answer;4;ip;1.2.4.5
Answer;2;ip;2.3.4.5

[download]
Results in: 
DEBUG: Flushing "complete" answer
{"Answer":[{"ip":"2.3.4.5"},{"ip":"2.3.4.5"}],"id":"2","host":"www.cnn
+.com"}
DEBUG: output remaining data ...
{"Answer":[{"ip":"3.4.5.6"},{"ip":"1.2.4.5"}],"id":"4","host":"www.goo
+gle.com"}
{"Answer":[{"ip":"1.2.3.4"}],"id":"1","host":"www.example.com"}
{"Answer":[{"ip":"3.4.5.6"}],"id":"3","host":"www.google.com"}
{"Answer":[{"ip":"2.3.4.5"}],"id":"2","host":"www.example2.com"}

[download]

Replies are listed 'Best First'.
Re^3: Memory utilization and hashes
by poj (Abbot) on Jan 18, 2018 at 14:30 UTC

    Same idea using one hash.

    #!/usr/bin/perl
use strict;
use warnings;
use JSON;

my %query  = ();
while (<DATA>) {
  chomp;
  next unless /\S/; # skip blank lines
  my ($s1,$n,$s2,$v2,undef) = split ';',$_,5;
  
  if ($s1 eq 'Query') {
    if (exists $query{$n}){
      # print and reuse
      output($n);  
    }
    $query{$n} = [$v2];
    
  } elsif ($s1 eq 'Answer') {
    push @{$query{$n}},$v2;
  }
}

# remaining
output($_) for keys %query;

sub output {
 my $n = shift;
 my $host = shift @{$query{$n}};
 print encode_json { id=>$n,host=>$host,ip => $query{$n} };
 print "\n";
}

    [download]
    poj
[reply]
[d/l]
      The OP did update his data here.

      A nit, but I think he wanted the anon array 'Answers' to be an array of hash refs.

      He was concerned about creating a hash too large for memory. :-)
      If the 'ids' were only 1 to 4, this wouldn't be a problem. But if the different 'ids' were in the thousands or so, I think that was what he what he wanted to avoid. You do reuse hash keys for this small example.

      There was only 1 repeated id in his data, and this is where it is reused.

      The test for 'exists' therefore tests true for this id only and output gets called. The rest of the output gets done outside the while loop (#remaining).

      Your code outputs (on his revised data with my ($s1,$n,$s2,$v2,undef) = split ';',$_,5; changed to my (undef,$s1,$n,$s2,$v2) = split ';',$_,5;): 

      {"ip":["2.3.4.5","2.3.4.5"],"id":"2","host":"www.cnn.com"}
{"ip":["3.4.5.6","1.2.4.5"],"id":"4","host":"www.google.com"}
{"ip":["1.2.3.4"],"id":"1","host":"www.example.com"}
{"ip":["3.4.5.6"],"id":"3","host":"www.google.com"}
{"ip":["2.3.4.5"],"id":"2","host":"www.example2.com"}

      [download]
      I coded a similar solution to you (only difference being an array of hashrefs instead of values alone). 
      {"id":"1","Answer":[{"ip":"1.2.3.4"}],"host":"www.example.com"}
{"id":"2","Answer":[{"ip":"2.3.4.5"},{"ip":"2.3.4.5"}],"host":"www.cnn
+.com"}
{"id":"3","Answer":[{"ip":"3.4.5.6"}],"host":"www.google.com"}
{"id":"4","Answer":[{"ip":"3.4.5.6"},{"ip":"1.2.4.5"}],"host":"www.goo
+gle.com"}
{"id":"2","Answer":[{"ip":"2.3.4.5"}],"host":"www.example2.com"}

      [download]
      My solution was: 
      #!/usr/bin/perl
use strict;
use warnings;
use JSON;

open my $fh, '<', 'file2' or die $!;

my %query;

while (<$fh>) {
    chomp;
    my (undef, $cat, $id, undef, $val) = split /;/;
    if ($cat eq 'Query') {
        if (%query) {
            output(\%query);
            %query = ();
        }
        $query{$id} = [$val];
    }
    else {
        push @{ $query{$id} }, {ip => $val};    
    }
}

output(\%query);

sub output {
    my $query = shift;
    my ($id) = keys %$query;
    my $host = shift @{ $query->{$id} };
    print encode_json({Answer => $query->{$id}, id => $id, host => $ho
+st}),"\n";
}

      [download]
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom