Re^2: Ampersands and sub speed

Perl/XS search engine library which uses the Java Lucene file format. Java Lucene uses method calls for everything down to writeByte(), so a big part of making things work in Perl is moving speed critical code into XS loops. But sometimes there's just no getting around calling a function or a method.

--
Marvin Humphrey
Rectangular Research ― http://www.rectangular.com

Comment on Re^2: Ampersands and sub speed

Replies are listed 'Best First'.
Re^3: Ampersands and sub speed by revdiablo (Prior) on Oct 14, 2005 at 22:16 UTC
But sometimes there's just no getting around calling a function or a method. And you're trying to run more than 1.4 million method calls per second? Update: I guess running one method call per byte, this would limit you to 1.4MB/s. Which isn't really all that bad, but could be a problem, I guess. I think it would be more fruitful to redesign the code not to call a method for every byte than it would to try and use a faster type of call, though.	[reply]
Re^4: Ampersands and sub speed by creamygoodness (Curate) on Oct 14, 2005 at 22:23 UTC
Say you have an index with a million documents in it. Someone searches for a term that hits 300,000 of them. Calculating the score for each one of them in Java Lucene might involve several tens of method calls. Now imagine a search with 5 terms. This stuff adds up. -- Marvin Humphrey Rectangular Research ― http://www.rectangular.com	[reply]
Re^3: Ampersands and sub speed by Errto (Vicar) on Oct 16, 2005 at 15:51 UTC
Just fyi, you should definitely check out Plucene if you haven't yet. Even if you can't use it directly (since the binary format of the index is not the same as Java Lucene), the code may be useful since they presumably have to solve the same problem you do.	[reply]
Re^4: Ampersands and sub speed by creamygoodness (Curate) on Oct 16, 2005 at 16:07 UTC
Good advice. I've been heavily involved in the Plucene project. This is like Plucene, but the indexer, at least, is 13 times faster, and it writes true Lucene-compatible indexes. I'm working on the search code. -- Marvin Humphrey Rectangular Research ― http://www.rectangular.com	[reply]