Re: creating and managing many hashes

An update: I spent the day reading-about/playing-with hashes, and decided to use them to the problem I had.

Satisfied with the results I now have at a constant 20% single-core utilization (approx), and a peak 1.5GB memory footprint.

$>perl pairs_v3.pl
Distinct products: 2083
Number of pairs (incl. duplicates): 4,336,806
Number of pairs (excl. duplicates): 2,168,403
Started processing at 14:04:49
Finished processing at 14:07:24
Finished writing to disk at 14:07:43
$>
[download]

PS: This is to determine the pairs only (haven't run the stats using the pairs yet which is the next step).

Comment on Re: creating and managing many hashes Download Code

Replies are listed 'Best First'.
Re^2: creating and managing many hashes by poj (Abbot) on Feb 19, 2018 at 10:27 UTC
This is to determine the pairs only That should take seconds, not minutes. `#!/usr/bin/perl use strict; my $t0 = time; my @f = map{ sprintf 'product_%04d',$_ } 1..2083; my $count = 0; open OUT,'>','pairs.txt' or die "$!"; while (my $x = shift @f){ for my $y (@f){ ++$count; print OUT "$count\t$x\t$y\n"; } } close OUT; my $dur = time-$t0; print "$count in $dur s\n"; # 19s on raspberryPI` [download] poj	[reply] [d/l]
Re^3: creating and managing many hashes by Gtforce (Sexton) on Feb 19, 2018 at 10:49 UTC
poj - that is amazing (took 3 seconds)!! Will do a post-mortem now, thanks a ton.	[reply]
Re^2: creating and managing many hashes by LanX (Saint) on Feb 19, 2018 at 10:29 UTC
I'm afraid, only telling us about "results" without showing code will only frustrate many and won't help you improve your skills. Cheers Rolf _{(addicted to the Perl Programming Language and ☆☆☆☆ :) Wikisyntax for the Monastery}	[reply]