Scanning each line is never scalable. What you need is text indexing (see f.ex. discussions 310891 249393), or simply to assign the whole, fully inflexed lexicon to a hash, so lookups become very cheap (at the expense of memory size though). If then memory becomes a problem, you might consider stuffing the lexicon into a database.