This makes things fairly easy to code, at least for relatively simple cases, but I very much doubt it would solve the performance problem.

For me, the real solution is using either a hash structure in memory (super fast if feasible) or an actual database such as MySql or MariaDB (or possibly even SQLlite, but that might be a bit more complicated with several files) with full support to indexed data access enabling fast processing.