Maybe you could try BerkeleyDB (which is a binding to the C library of the same name).  As you essentially only have simple key value associations, and it's exactly tailored for such kind of problems, it might be more memory efficient than SQLite. The current version of Berkeley DB supports table sizes of up to 256 TB.