in reply to Writing inverted index code in perl might be overkill
in thread Writing a Search Engine in Perl?

Eh nice link I made (if that is the right word in English anyway) ...

Why do you think that writing reverse index in Perl would be overkill? And in what meaning it would be an overkill?

Sure C/Perl combination might be considered - I could finally put C/C++ knowledge gained on advanced school to practical use ...

  • Comment on Why? - Writing inverted index code in perl might be overkill

Replies are listed 'Best First'.
Re: Why? - Writing inverted index code in perl might be overkill
by dpavlin (Friar) on Aug 19, 2005 at 17:18 UTC
    Only down-side to perl only version is speed. Of course, it depends on size of your input data. However, on my laptop I have more data that I want to index than any perl-only solution really can handle (over 20Gb in various formats).

    I have some expiriences with WAIT (and some pending patches at http://svn.rot13.org/~dpavlin/svnweb/index.cgi/wait/log/trunk/ ), swish-e, Xapian (another great engine which updated perl bindings few days ago). I also experimented with CLucene perl bindings and finally ended with HyperEstraier.

    I would suggent to make list of requirements of search engine and then select right one. My current list include:

    • full text search
    • filter results by attributes (e.g. date, category...)
    • ability to update index content while running searches on it
    • wildcard support (or substring, even better!)
    • acceptable speed on projected amount of data
    Last point influence choice very much. I would go with Plucene if data size is small enough (or only for prototyping).

    Writing good parsers and analyzers for input formats (do you want to rank bold words more than surround text?) and font-end is hard enough without writing own reverse index implementation, especially since some very good allready exist.


    2share!2flame...
[reply]
      In my experience, Plucene was not very good at handling your third requirement, "ability to update index content while running searches on it." The code that handles file locking is prone to die instead of wait. Not good for live websites. The following ASCII depicts my expression upon discovering this:

      8-[

      YMMV
[reply]

In Section Seekers of Perl Wisdom