thanks for your reply! i start with 100 points each with x,y,z coordinates. 

4.941 32.586 -1.772
15.354 22.823 10.556
-0.495 12.345 98.234
...


$block_size = 1000;
$counter = 0;
%small_hash = ();
foreach @triplet of the 100 points{
     $counter++;
     my @matrix = ortho(@triplet);
     foreach $point in the 100 points{
           my $key = binning(@matrix * $point);
           push @{$small_hash{$key}},@matrix;
     }
     #binning causes many points to fall in the same grid,
     #thus the array of values (matrices) for each key
     if($counter % $block_size == 0){
           write out %small_hash; 
           %small_hash = (); #frees the memory
     }
     
}

# now merging small hash files
foreach $key (sort keys %master_hash){#keys for master_hash stored
     loop thru all small files{
          while(<FILE>){
               if($_ =~ /^$key/){
                    write values to the master output file;
                    last;
               }#if
          }#while
     }#go thru files
}

[download]
so that's roughly the structure (actual code too long to post here). the first half is really fast to compute, but once i start merging files it takes forever. thanks a lot!

In reply to Re^2: how to merge many files of sorted hashes? by andromedia33
in thread how to merge many files of sorted hashes? by andromedia33

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.