Good point. At this stage of the process, the data is being read from a text file. It is ultimately put into a data base but not right away. In the database itself, the fields to be cross referenced are not in a form suitable for a join or subselect. In fact, that was one of the problems with the original system. The categorization performed after the database was loaded was unacceptably slow (due to the need to match subsets) and somewhat error prone (due to poor system design).

The data in the data base is part of a legacy system and it is not practical to alter its format as too much else depends on it. In terms of operations, I found it more efficient to perform the categorization before loading it into our data base. This allows me to transform the data from the input file into a form that lends itself to the categorization scheme and then transform the result into the legacy format with the categorizations already in place. This reduces the load operation to a simple insert/update rather than an insert/update followed by modification.

PJ
unspoken but ever present -- use strict; use warnings; use diagnostics; (if needed)

In reply to Re^2: Dynamic Lookups on SDBM Files by periapt
in thread Dynamic Lookups on SDBM Files by periapt

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.