Re^9: Finding the size of a nested hash in a HoH

Your benchmark is still fatally flawed.

The major cost of using keys, is the requests for extra memory from the OS in order to build the list of keys. This is a one-off cost, that only occurs the first time the process requests that memory from the OS. By running your code in a loop (using Benchmark), you are amortising that costs over many re-runs, thereby reducing its significance. And by running both methods in the same benchmark, your are further amortising the penalty of that memory allocation over the each method runs which don't need it, thus further skewing the results.

This is a Windows-specific benchmark (due to the use of tasklist.exe in memUsed()) but should be readily adaptable to *nix:

On my system, for 1 million keys, for / keys required 56MB of extra memory and so is 10x slower than while / each:

c:\test>each-keys-b -M1=keys -M2=pairs -N=1e6
hash built, starting timer
Took 11.663000 and 56.145MB extra memory for 1000000 using keys/pairs 
+method

c:\test>each-keys-b -M1=each -M2=pairs -N=1e6
hash built, starting timer
Took 1.296000 and 0.020MB extra memory for 1000000 using each/pairs me
+thod
[download]

For 5 million keys, for / keys required 228MB of extra memory and so is 40x slower than while / each:

c:\test>each-keys-b -M1=keys -M2=pairs -N=5e6
hash built, starting timer
Took 280.350000 and 228.652MB extra memory for 5000000 using keys/pair
+s method

c:\test>each-keys-b -M1=each -M2=pairs -N=5e6
hash built, starting timer
Took 6.613000 and 0.020MB extra memory for 5000000 using each/pairs me
+thod
[download]

For 8 million keys, for / keys required 340MB of extra memory and so is 80x slower than while / each:

c:\test>each-keys-b -M1=keys -M2=pairs -N=8e6
hash built, starting timer
Took 899.040000 and 343.402MB extra memory for 8000000 using keys/pair
+s method

c:\test>each-keys-b -M1=each -M2=pairs -N=8e6
hash built, starting timer
Took 11.112000 and 0.023MB extra memory for 8000000 using each/pairs m
+ethod
[download]

And at these sizes of hash, my machine has not yet moved into swapping. When I retire tonight, I'll leave the machine running on 10 million keys which will cause swapping and I'll report the timings tomorrow.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Comment on Re^9: Finding the size of a nested hash in a HoH Select or Download Code

Replies are listed 'Best First'.
Re^10: Finding the size of a nested hash in a HoH by remiah (Hermit) on Nov 12, 2011 at 01:56 UTC
Replacing tasklist.exe with ps, your code worked with my machine and it shows while/each is faster and smaller. 1. About Speed I noticed from your code, I was very unfair to while/each because I lack value assignment with foreach/keys test function. When I change Benchmark test function from `foreach my $k (keys %h){ }` [download] to `foreach my $k (keys %h){ my $v=$h{$k}; }` [download] , benchmark shows while/each becomes faster Read more... (1340 Bytes) 2. About memory With ps command, while/each loop shows very very small memory usage. Memory allocation of "keys" is significant. `$./025-6.pl -M1=keys -M2=pairs -N=1e6 hash built, starting timer Took 3.931463 and 39.000MB extra memory for 1000000 using keys/pairs m +ethod $./025-6.pl -M1=each -M2=pairs -N=1e6 hash built, starting timer Took 2.434757 and 1.000MB extra memory for 1000000 using each/pairs me +thod` [download] It seems I miss something with GTop.	[reply] [d/l] [select]
Re^11: Finding the size of a nested hash in a HoH by BrowserUk (Patriarch) on Nov 12, 2011 at 02:46 UTC
I noticed from your code, I was very unfair to while/each because I lack value assignment with foreach/keys test function. When I change Benchmark test function from `foreach my $k (keys %h){ }` [download] Look at the code again carefully. When the command line argument `-M2=pairs` the code run for the for / keys test is: `else { if( $M2 eq 'pairs' ) { for my $k ( keys %hash ) { my $v = $hash{ $k }; ++$count; } }` [download] Ie. This is equivalent to the while / each case for `-M2=pairs` of: `if( $M1 eq 'each' ) { if( $M2 eq 'pairs' ) { while( my( $k, $v ) = each %hash ) { ++$count; } }` [download] To perform the test for both cases where only the keys are accessed, use a command line argument of `-M2=keysonly` (or any value other than `pairs`). The code that gets run for two methods in this case are: `else { while( my $k = each %hash ) { ++$count; } }` [download] and `else { for my $k ( keys %hash ) { ++$count; } }` [download] Again, equivalent code. Thanks for confirming my premise. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l] [select]