Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Re^2: Memory utilization and hashes

by bfdi533 (Friar)
on Jan 17, 2018 at 22:26 UTC ( [id://1207445]=note: print w/replies, xml ) Need Help??


in reply to Re: Memory utilization and hashes
in thread Memory utilization and hashes

Since I need all of the query and answers info on one line in the output, yes, I need to collect them up until I have all of the answers.

here is a more closely working example of the code. I was trying to keep it simple and focus on the memory usage of the hash but here we are.

#!/usr/bin/perl use warnings; use strict; $|++; use JSON; my $l; my @vals; my $json; my %pairs; my %pind; my %flush; while (<DATA>) { $l = $_; chomp $l; @vals = split /;/, $l; if ($vals[0] =~ /Query/) { if (! $pairs{$vals[1]}) { $pind{$vals[1]} = 0; } if (!defined $flush{$vals[1]}) { $flush{$vals[1]} = " "; } elsif ($flush{$vals[1]} ne $vals[1]) { $json = encode_json $pairs{$vals[1]}; print "DEBUG: Flushing \"complete\" answer\n"; print $json."\n"; delete $pairs{$vals[1]}; $flush{$vals[1]} = $vals[1]; $pind{$vals[1]} = 0; } $pairs{$vals[1]}{$vals[2]} = $vals[3]; $pairs{$vals[1]}{id} = $vals[1]; } elsif ($vals[0] =~ /Answer/) { $pairs{$vals[1]}{$vals[0]}[$pind{$vals[1]}++]{$vals[2]} = $val +s[3]; } } print "DEBUG: output remaining data ...\n"; foreach my $key (keys %pairs) { $json = encode_json $pairs{$key}; print $json."\n"; } __DATA__ Query;1;host;www.example.com Answer;1;ip;1.2.3.4 Query;2;host;www.cnn.com Query;3;host;www.google.com Answer;2;ip;2.3.4.5 Answer;2;ip;2.3.4.5 Query;4;host;www.google.com Answer;4;ip;3.4.5.6 Answer;3;ip;3.4.5.6 Query;2;host;www.example2.com Answer;4;ip;1.2.4.5 Answer;2;ip;2.3.4.5
Results in:
DEBUG: Flushing "complete" answer {"Answer":[{"ip":"2.3.4.5"},{"ip":"2.3.4.5"}],"id":"2","host":"www.cnn +.com"} DEBUG: output remaining data ... {"Answer":[{"ip":"3.4.5.6"},{"ip":"1.2.4.5"}],"id":"4","host":"www.goo +gle.com"} {"Answer":[{"ip":"1.2.3.4"}],"id":"1","host":"www.example.com"} {"Answer":[{"ip":"3.4.5.6"}],"id":"3","host":"www.google.com"} {"Answer":[{"ip":"2.3.4.5"}],"id":"2","host":"www.example2.com"}

Replies are listed 'Best First'.
Re^3: Memory utilization and hashes
by poj (Abbot) on Jan 18, 2018 at 14:30 UTC

    Same idea using one hash.

    #!/usr/bin/perl use strict; use warnings; use JSON; my %query = (); while (<DATA>) { chomp; next unless /\S/; # skip blank lines my ($s1,$n,$s2,$v2,undef) = split ';',$_,5; if ($s1 eq 'Query') { if (exists $query{$n}){ # print and reuse output($n); } $query{$n} = [$v2]; } elsif ($s1 eq 'Answer') { push @{$query{$n}},$v2; } } # remaining output($_) for keys %query; sub output { my $n = shift; my $host = shift @{$query{$n}}; print encode_json { id=>$n,host=>$host,ip => $query{$n} }; print "\n"; }
    poj
      The OP did update his data here.

      A nit, but I think he wanted the anon array 'Answers' to be an array of hash refs.

      He was concerned about creating a hash too large for memory. :-)
      If the 'ids' were only 1 to 4, this wouldn't be a problem. But if the different 'ids' were in the thousands or so, I think that was what he what he wanted to avoid. You do reuse hash keys for this small example.

      There was only 1 repeated id in his data, and this is where it is reused.

      The test for 'exists' therefore tests true for this id only and output gets called. The rest of the output gets done outside the while loop (#remaining).

      Your code outputs (on his revised data with my ($s1,$n,$s2,$v2,undef) = split ';',$_,5; changed to my (undef,$s1,$n,$s2,$v2) = split ';',$_,5;):

      {"ip":["2.3.4.5","2.3.4.5"],"id":"2","host":"www.cnn.com"} {"ip":["3.4.5.6","1.2.4.5"],"id":"4","host":"www.google.com"} {"ip":["1.2.3.4"],"id":"1","host":"www.example.com"} {"ip":["3.4.5.6"],"id":"3","host":"www.google.com"} {"ip":["2.3.4.5"],"id":"2","host":"www.example2.com"}
      I coded a similar solution to you (only difference being an array of hashrefs instead of values alone).
      {"id":"1","Answer":[{"ip":"1.2.3.4"}],"host":"www.example.com"} {"id":"2","Answer":[{"ip":"2.3.4.5"},{"ip":"2.3.4.5"}],"host":"www.cnn +.com"} {"id":"3","Answer":[{"ip":"3.4.5.6"}],"host":"www.google.com"} {"id":"4","Answer":[{"ip":"3.4.5.6"},{"ip":"1.2.4.5"}],"host":"www.goo +gle.com"} {"id":"2","Answer":[{"ip":"2.3.4.5"}],"host":"www.example2.com"}
      My solution was:
      #!/usr/bin/perl use strict; use warnings; use JSON; open my $fh, '<', 'file2' or die $!; my %query; while (<$fh>) { chomp; my (undef, $cat, $id, undef, $val) = split /;/; if ($cat eq 'Query') { if (%query) { output(\%query); %query = (); } $query{$id} = [$val]; } else { push @{ $query{$id} }, {ip => $val}; } } output(\%query); sub output { my $query = shift; my ($id) = keys %$query; my $host = shift @{ $query->{$id} }; print encode_json({Answer => $query->{$id}, id => $id, host => $ho +st}),"\n"; }

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1207445]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others admiring the Monastery: (2)
As of 2024-04-16 14:47 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found