in reply to Re^3: Apparently strange beahavior that blocks at Filehandle reading
in thread Apparently strange beahavior that blocks at Filehandle reading

Ok I'll show you the structure:

The attention is on 'COMB' key that reference the hash with 180000 keys of about 20-25 chr. The program must increase all the "lower level" key {Rit} for each "while" cicle that read line by line from the previous discussed Files (9000+ lines).

I know its a huge work, but who has an idea on how to make it faster? at this time it processes the 180000 values for 50 times (~9 million) in about 12 secs (the whole work could take about 2160 secs)

Thank you again to all of you! 


%DB_STATS = (

   'COMB' => {
              '1,2,4,6,9,11,13,15,16,17' => {
                                               'Rit' => 0,
                                               'Usc' => 0
                                            },
              '1,2,4,6,9,11,13,15,16,18' => {
                                               'Rit' => 0,
                                               'Usc' => 0
                                            },
              '1,2,4,6,9,11,13,15,16,19' => {
                                               'Rit' => 0,
                                               'Usc' => 0
                                            },
              'And 180_000 more Keys...' => {
                                               'Rit' => 0,
                                               'Usc' => 0
                                            },
             },

    'Other keys NOT of our interest',
    ...,
    ...,

);

[download]

Replies are listed 'Best First'.
Re: About Data Structure and elaboration time (6x faster)
by BrowserUk (Patriarch) on Jun 17, 2011 at 14:15 UTC

    Instead of dereferencing the entire multilevel structure each time, if you create an array of references to the scalars you are incrementing, then you can iterate over that array and increment the values in the nested hash in 1/6th the time:

    #! perl -slw
use strict;
use Time::HiRes qw[ time ];

my %DB_STATS;
$DB_STATS{ COMB }{ $_ } = { Rit=>0, Usc=>0 } for 1 .. 18e5;

my $start = time;
++$DB_STATS{ COMB }{ $_ }{Rit} for 1 .. 18e5;
printf "Full deref took %.3f seconds\n", time() - $start;

my @Rits;
$Rits[ $_ ] = \$DB_STATS{ COMB }{ $_ }{Rit} for 1 .. 18e5;

$start = time;
++$$_ for @Rits;
printf "AoR via alias took %.3f seconds\n", time() - $start;

print $DB_STATS{COMB}{1}{Rit};

__END__
C:\test>910049
Full deref took 1.511 seconds
AoR via alias took 0.247 seconds
2

    [download]
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]

In Section Seekers of Perl Wisdom