May I ask *why* swapping to SQL is not an option? (Even if you have a good reason, I'd like to know -- sometimes these types of questions start excellent threads.)

If it's a full-blown SQL database that you'd like to avoid, you might want to take a gander at SQLite. It's an SLQ database, but it's minimalistic, easy to install, and all data lives in one file on the filesystem. Pretty cool for those 'too-big-but-not-big-enough' problems.