Full Text Search

skullbowl has asked for the wisdom of the Perl Monks concerning the following question:

I have 30 volumes of conference articles, each volume consists of about 200 to 400 articles and each article is about 5-11 pages long. These articles are meant for users for reference on a local server without a search engine. I have scanned the articles using Adobe Writer but users are not able to do a full text search in pdf format. Can I write a perl script for users to perform a full text search? Anywhere I can download some sample codes to test?

Thanks.

use Text::Soundex;
print soundex ("Skullbowl");
[download]

RESULT : S414

Comment on Full Text Search Download Code

Replies are listed 'Best First'.
Re: Full Text Search by tinman (Curate) on May 13, 2001 at 22:30 UTC
It sounds like a fairly big task in itself.. I thought a bit about it and perhaps you could try one of the following approaches... Firstly, if you need full text searching, its probably wise to build indexes first... searching all the PDF files on your server is going to take far too long, and you could run into scalability and concurrency issues.. So the solution would probably be to periodically convert the textual content of the PDF files into a text file, and index that text file... Text indexing is pretty well understood, and its fast, moreover, so this would probably be the most efficient method.. Just provide a link to the real PDF file in your search results Another thing you could try is to convert the PDF to HTML (see the links below).. Have a converter program product HTML output, and then run an indexing tool like ht://Dig or something on the HTML output.. Then, your users can read the output in HTML, and if desired download the PDF format document. These are some links I found that might make your life easier, mostly to do with PDF conversion from one form or another, but there are some links on how to convert PDF to other formats as well... PDF tools Perl modules for PDF conversion HTH	[reply]
Re: Full Text Search by Hero Zzyzzx (Curate) on May 14, 2001 at 00:13 UTC
I've done some thinking on this. This is how I'm going to approach this, with a focus around ease of implementation. Here's one way to approach it: Use `pdftotext`, which I believe is provided by xpdf to get the text out of each pdf (this works for compressed (or "optimized" in adobe speak) pdfs too). Put this text into a mySQL database. (or dump it out to the filesystem and index with htdig.) Then either use mySQLs fulltext index feature, (which is limited in it's features, but still damn fast and pretty good, too) to create your search index or create your own mySQL search query.	[reply] [d/l]
Re: Re: Full Text Search by Hero Zzyzzx (Curate) on May 14, 2001 at 17:46 UTC
In my research around this, I remember there being a search engine project on SourceForge that had code to dump text from pdfs using pdftotext. Try searching there.	[reply]
Re: Full Text Search by srawls (Friar) on May 13, 2001 at 22:08 UTC
Sounds like a big task. Well, I searched CPAN and found TEXT::SEARCH. I've never used it, so there may be better modules than that, though. I would suggest just browsing at CPAN for a little bit. Best of luck. The 15 year old, freshman programmer, Stephen Rawls	[reply]