The Berkeley DB seems to meet you specification perfectly it's not a straight forward DB by any means however, check out BerkeleyDB, very fast, key / value pairs, db_dump and db_load for bulk loading, full ACID transactions etc... however you would have to host the DB environment on the same server, it doesn't support network access as it's an embedded database. The hash structure would suit your needs. It does fail on the compact storage aspect as a DB Hash is larger than the original data, but that's the trade off for fast access.

As suggested previously an LDAP server (Net::LDAP) might also suit your needs.