As far as I can see the intersection problem is On*Ologn which make scaling it problematic.
How does 312ms sound for a two word query in a DB containing 15,000 words in 5000 documents (ave:554 words/doc; 2.7million word-document pairs?
678848=# select
678848-# "wordFromId"( "word-id" ),
678848-# "docnameFromId"( "doc-id" ),
678848-# posns
678848-# from "word-doc"
678848-# where
678848-# "word-id" in (
678848(# "idFromWord"( 'aachen' ),
678848(# "idFromWord"( 'zwitterions' )
678848(# )
678848-# and
678848-# "doc-id" in (
678848(# select "docIdsContaining"( "idFromWord"( 'aach
+en' ) )
678848(# intersect
678848(# select "docIdsContaining"( "idFromWord"( 'zwit
+terions' ) )
678848(# )
678848-# order by
678848-# "word-id", "doc-id"
678848-# ;
wordFromId | docnameFromId | posns
-------------+---------------+-------
aachen | document_223 | 1723
aachen | document_940 | 1903
aachen | document_1778 | 273
aachen | document_1897 | 3163
aachen | document_3990 | 817
aachen | document_4407 | 4736
zwitterions | document_223 | 3072
zwitterions | document_940 | 3504
zwitterions | document_1778 | 664
zwitterions | document_1897 | 4186
zwitterions | document_3990 | 355
zwitterions | document_4407 | 1459
(12 rows)
678848=# select count( * ) from "word-doc";;
count
---------
2719282
(1 row)
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.