Just fyi, you should definitely check out Plucene if you haven't yet. Even if you can't use it directly (since the binary format of the index is not the same as Java Lucene), the code may be useful since they presumably have to solve the same problem you do.