in reply to Dynamic Lookups on SDBM Files

It is not really apparent n your post what the actual data and looks look like, but would it make more sense to dump this data in a database like mysql or postgres and then have the lookups defined as SQL? That way (of course depending what the actual lookup needs are) you could have one SQL statement for each type of lookup. I am thinking something like what you state is your cross reference list that changes may easily fit into a SQL join or subselect. Could you post an example of the dataset and the types of lookups you are doing (with expected results from the example set). This should give us a better understanding of the actual problem and you better output from us.


-Waswas

Replies are listed 'Best First'.
Re^2: Dynamic Lookups on SDBM Files
by periapt (Hermit) on Aug 04, 2004 at 12:23 UTC
    Good point. At this stage of the process, the data is being read from a text file. It is ultimately put into a data base but not right away. In the database itself, the fields to be cross referenced are not in a form suitable for a join or subselect. In fact, that was one of the problems with the original system. The categorization performed after the database was loaded was unacceptably slow (due to the need to match subsets) and somewhat error prone (due to poor system design).

    The data in the data base is part of a legacy system and it is not practical to alter its format as too much else depends on it. In terms of operations, I found it more efficient to perform the categorization before loading it into our data base. This allows me to transform the data from the input file into a form that lends itself to the categorization scheme and then transform the result into the legacy format with the categorizations already in place. This reduces the load operation to a simple insert/update rather than an insert/update followed by modification.

    PJ
    unspoken but ever present -- use strict; use warnings; use diagnostics; (if needed)
[reply]

In Section Seekers of Perl Wisdom