Stemming, eh? Hmmm...interesting concept. I'll research it.

Overall, the total size of my current content file (even if I add in all the html template pages) is just over a meg. This generates over 100 pages of content at my site currently.

If the content size was, say, 10 times greater, I would agree with you as searching a 10 meg file would take time. I think for the first version I will probably not stem the content...or at least not until I have a better grasp on the specifics of it. Also, by leaving things in this way, anybody using whatever db engine they wanted to use could do searches on the data, including phrase searching say like thru DBI, DBD::AnyData, and SQL::Statement. ;D

Thanks for that bit of info about stemming...I will look into it.

======================
Sean Shrum
http://www.shrum.net

I have posted a node enquiring into stemming algorithms in Perl here, in particular, the Porter algorithm employed in Lingua::Stem, which may be of interest to you.

 

The first being the roll-your-own solution, which while being prehaps the most involved, that guaranteed to be most customised to your needs :-) For this, I would advise either using an indexing module such as my own Local::SiteRobot or WWW::SimpleRobot (the author of which has updated based on patches submitted). In following this path, I would advise you to do a bit of research and code audit of some of the existing search and indexing solutions - For example, a flat file text storage base simply won't scale, the better option, employed by most other scripts of this nature, is tied hash or DBM file storage. For content and meta-tag following, some of the existing modules such as HTML::TokeParser will shorten your development time and programmer migraines immensely.

The other way is to explore some of the existing solutions, one of the better options which I found when I was looking into this issue was the Perlfect Search script which offers HTTP and file system indexing, PDF text extraction, meta-tag following, ranking support and template-system output. I haven't as yet had a chance to set this up on one of my development boxes and while a preliminary code review has found a couple of quirky style and indexing issues, this package looks fairly solid.

Good luck ... And feel free to /msg me if you have any questions.

perl -e 's&&rob@cowsnet.com.au&&&split/[@.]/&&s&.com.&_&&&print'

Meta-tag following?!?!? Nah! Nothing that complex.

My only concern at present would be to get the links out of the document for following. Basically, anything that can be seen publically in the document is what I want to index on. This will allow me to have pages on my site that don't have referring links to them and will not be indexed for general public viewing.

Some "off-the-top-of-my-head" planning
=====================================
What I could use (and I think I saw in CPAN somewhere) is a mod that extracts all the links on a specified URL. I could then check, via a loop, that the link string begins/contains/ends with a user defined string (ie: "http://www.shrum.net") to keep tranversals under control.

I could then search a hash where I store links in to make sure that duplicates link entries aren't created and introduce the new links into the hash.

At the same time, the contents of the page will be stripped of their HTML (I think I saw a mod that does this too at CPAN) and the raw document data will be stored into the hash associated with it's cooresponding URI.

I guess I would need a INDEXED flag for each URI to indicate that the URI has been visited and it's page content copied. This would be done last to visit any pages that had links to them but were skipped during the first tranversal.

Once all the INDEXED flags are taken care of, the data in the hash would be written to a flat-file that could then be searched with whatever db engine you wanted.

==========================

This is just my first-pass line of thinking. I will sleep on it tonight and digest about it some more tomorrow. If you have any suggestions, I am all ears.

Thanx for the help so far!

======================
Sean Shrum
http://www.shrum.net

I write in further support of Perlfect Search,
It will let you create a list of excluded folders/files, that won't get indexed,
and just because it has complex features that you don't need (at the moment),
doesn't mean you have to use them.

I use Perlfect on one of my sites, and it's very good.
The only minus point, for me, is that is doesn't support live searches, so it's necessary to create a cron job to regularly re-index the site.

The problem there is that my site uses nested html documents...the URI will be a perl script call that will take the following:

Page = General site identity document
Table = Content header (table results, document abstract info, etc.)
Record = Document content (tabled data, documents, etc.)

...and nest them in the following order: Record -> Table -> Page

Just searching the files would create a list of all the words at the site however it will not take into consideration the layout of the files and their nesting order...the URI is the key.

======================
Sean Shrum
http://www.shrum.net

You could expand the filename. I am assuming when you say nested you mean subdirectories.

Let's say your document root is /usr/local/apache/htdocs

#!/usr/bin/perl
use File::Find;

my $filedir  = '/usr/local/apache/htdocs';
my $baseurl = "http://someplace.com/shrum";

my @docs  = ();
sub process_file {
   return if -d;    # Skip directories.
   push @docs, [$File::Find::dir, $_];
  
}

find(\&process_file, ($filedir) );


foreach $doc (@docs){

    $doc->[0]=~s/$filedir/$baseurl/o;
    #[0] is dir [1] is filename.
    print "URL is ".$doc->[0].'/'.$doc->[1],"\n";
}
[download]



-Lee

"To be civilized is to deny one's nature."

A few other points to clarify...

Some of the site content is dynamically generated through perl scripts. Using a robot would allow me to index this content as well. As Shotgunefx pointed out, a low-level file dump would not produce the results I am looking for.

In any event, the script I am thinking of making would allow for things like multi-domain searches (like if you had "http://www.mysite.com" and "http://search.mysite.com" and etc.) but will also allow you to index remote sites as well (if you wanted to; probably piss off the domain owner if they found out).

If this sounds like a good idea and you happen to have some first-hand experience with a mod(s) that might help me achieve my proposed end-result, let me know. Also, if you have sample code to invoke those mods, please pass that along too! Thanx.

======================
Sean Shrum
http://www.shrum.net

Generically indexing dynamic pages might be tricky. For a start on the indexing, you may want to look at merlyns parallel link checker column.

-Lee

"To be civilized is to deny one's nature."

Maybe not directly a Perl solution, but you might want to check SWISH-E