I think that a more efficient database would be your best speed boost. One of the dbm family might be enough from the way you describe the processing.

Speeding up the slow part is often the best strategy.

After Compline,
Zaxo