Good point. At this stage of the process, the data is being read from a text file. It is ultimately put into a data base but not right away. In the database itself, the fields to be cross referenced are not in a form suitable for a join or subselect. In fact, that was one of the problems with the original system. The categorization performed after the database was loaded was unacceptably slow (due to the need to match subsets) and somewhat error prone (due to poor system design).

The data in the data base is part of a legacy system and it is not practical to alter its format as too much else depends on it. In terms of operations, I found it more efficient to perform the categorization before loading it into our data base. This allows me to transform the data from the input file into a form that lends itself to the categorization scheme and then transform the result into the legacy format with the categorizations already in place. This reduces the load operation to a simple insert/update rather than an insert/update followed by modification.