Re^2: Most efficient record selection method?

Ok.

Variants, = mixture of values contained in each field.

Sample records = Live records containing these values

I don't care about the combination, but we do need to have the minimum number of records so:


[A1,B1],[A1,B2],[A2,B1],[A2,B2]  (variants over 2 fields)
[download]

could be resolved in 3 records as you have said:

record 1 - contains [A1,B1]
record 2 - contains [A1,B2]
record 3 - contains [A2,B1]
[download]

HOWEVER, if there are 2 records that contain:

record 1 - [A1,B1]
record 2 - [A2,B2]
[download]

Then these need to be identified and used in preference to the first combination as that will be the minimum number of records I need to represent all the variants in each of the fields.

This gets more complicated as the number of fields and files being interrogated increases

Comment on Re^2: Most efficient record selection method? Select or Download Code

Replies are listed 'Best First'.
Re^3: Most efficient record selection method? by ELISHEVA (Prior) on Feb 13, 2009 at 13:48 UTC
Ok. Here is a start on an algorithm. The sample code contains two fake files (represented by arrays of lines) to demonstrate how to maintain state as we pass from one file to the next. It also contains a very simple optimizer that can eliminate lines that add only one new value in favor of lines that add two or more new values at once. However, that optimization is not going to find the actual smallest sample if the number of selected fields is greater than 2. It is also possible that we might find a record with two new values and then later on a record with the same two values plus a new value in a third field. You will need to replace the simple optimizer with a more general purpose optimizer. Alas, coming up with a general solution is more work than I have time for. Read more... (3 kB) Best, beth Update: Fixed small bug	[reply] [d/l]

Replies are listed 'Best First'.

Re^3: Most efficient record selection method?
by ELISHEVA (Prior) on Feb 13, 2009 at 13:48 UTC

However, that optimization is not going to find the actual smallest sample if the number of selected fields is greater than 2. It is also possible that we might find a record with two new values and then later on a record with the same two values plus a new value in a third field. You will need to replace the simple optimizer with a more general purpose optimizer. Alas, coming up with a general solution is more work than I have time for.