Keep them separate only if you need to do complex things dealing with them in distinction from one another. It sounds like generally you need both pieces of data, so it makes sense to keep them together in light of the extra resources required for another entire hash. I suggest accessing the array elements though using constants, so you'd have:

use constant SCORE => 0;
use constant NUM_MATCHES => 1;
my $num = $various_scores{$this_document}[NUM_MATCHES];
[download]

Adding further entries, which should occur fairly infrequently, is merely a matter of adding new constants.

• there would be a "final score" to be computed through some (possibly simple) combination of "feature scores", such as summing, weighted average, etc.
• there may be cause to add more "feature scores" to supplement the two that you described

I think these both tend to disfavor an approach where you have separate/parallel hashes, because coding the existing method, and adding extensions later, requires larger, more tedious amounts of script. Much easier to combine numbers that happen to be in one array already, and much easier to just push more numbers onto the array as needed.

If you're worried about the potential obscurity of using just an array index (0, 1, ...) to mean "single-term frequency score", "combined-term score", ... well, there are easy ways to keep it clear and explicit (#COMMENT IT!) -- at worst, you could use HoH to store the numeric values of each type for each doc, but that seems unnecessary in this sort of case.

BTW, another feature that is commonly used in searches is the relative "distinctiveness" of search terms: given some basis for knowing the a priori likelihood of each term in general usage, assign more or less weight to its occurrence in a document; for example, if the search terms are "ionized charged", documents that contain only the first should probably score higher than those that contain only the second, and the relative weights of these two terms is inversely proportional to their respective frequencies in some collection of typical text data.

I measured 4352k for the two-hash case and 4944k for the single hash of arrays. It makes sense to me that the hash of arrays costs more because of all those extra references.

What data structure makes sense for a search engine? I'm not sure either of these do, if you are scoring hundreds of thousands of pages. You don't want to put all those scores and page references in memory and then sort them.

What might make sense is a heap, in which you can keep the top N best scores partially sorted with a logarithmic insertion time. There's a Heap module for a starting point.

Using parallel hashes has the advantages of

• a) taking less memory

  2 hashes -- 100_000 docs/10 char names/2 numeric values ~= 14MB

  Hash of 2-element arrays -- 100_000/10-char names/2 numeric values ~= 23MB.

• The named hashes indexed by doc.name are somewhat clearer than a docs hash indexed by doc.name, sub-index by mysterious numbers-- though that can easily be alleviated by using

  use constant TOTAL_WORD_SCORE => 0;
  use constant NUMER_OF MATCHES => 1;
  [download]

You pay the price for all those 2-element arrays. However, if the types of information you are going to store about each document is likely to extend, then there are rapidly diminishing returns from using parallel hashes.

By the time you get to 4 hashes -v- 1 hash of 4-element arrays, the penalty of duplicating the hash keys comes to bear with the former taking ~= 25MB and the latter ~= 30MB.

So it's easy to see that from the extensibility point of view, the HoA's comes out a winner pretty quickly and you should probably only consider the former if you're pretty certain that you won't need to extend and/or you're really up against it memory wise.

my $search_string = $cgi->param('search_string');
my @display_these =
    map { $_->[1] } # Strip off the ST
    sort { $a->[0] <=> $b->[0] } # Do the actual sort
    grep { $_->[0] > 0 } # Remove anything that scores a 0
    map { [ $_->score($search_string), $_ ] } # Create the modified ST
    @document_objects;
[download]

Plus, you left out the part where this is really a 3-level structure, not a 2-level structure. You probably don't think of it that way, but you are de-referencing one level before you even get to the code posted above.

------
We are the carpenters and bricklayers of the Information Age.

Don't go borrowing trouble. For programmers, this means Worry only about what you need to implement.