My advice is this: See postgreSQL and HTML::Mason.

Mason has nice builtin caching that allows you to build your pages dyanamically from your database, then keep the page around for better performance.

Thanks for the comments, I was thinking about Mason, the trouble is I havent really done any work with it before, and doesnt it require mod_perl? Also the thought of OO Perl is a little worrying... but i'll definately think about it. Caching is a good idea even if I dont go for Mason.

cheers,
Tom

If you are afraid of using OO Perl, using an unknown module or mod_perl, perhaps you shouldn't do the task of creating a directory site. Or at least not in Perl, perhaps you are more fluently in other languages. But your Perl skills don't seem to be evolved enough to handle a big task as you described.

Abigail

Altho' perl can most definately do what you want, it may not be the best tool for you.

What are the goals of the resedign? just saying improved maintaince is not really enuff... what you need to do is start at the top, get agreed goals, and break down the project pieve (pun intended) by piece into smaller chunks.

For each small chunk, you can say it will take me 2-3 weeks to complete a production quality application or peice. From this data you can quantify to management how long it will take, and you can "Manage their expectations (tm)", which is always a good thing.

So, in short

Just wondering if anyone could advise me on suitable ways of developing...

Planning.

CodeJunkie,
I'm kind of needing this too and I've been working on it in my spare time to update a site for work... unfortunatly there doesn't seem to be anything out there really that is much help for this specific problem.

I'm going to stay away from code that generates the pages. The problem with that approach is that it still creates a mess especially in my experience working with people who thought they could simply update the HTML pages which of course got everything out of sync. :) YMMV however if you are working on it on your own or with people that understand they need to use the update tool.

I'm planning on using MySQL and building it like a normal web app, using CGI, HTML::Template and possibly CGI::Application, etc. My suggestion is use a templating system and make it modular. I'll share my stuff but at the moment there isn't a whole lot to share (and actually I may have to write it in PHP since my boss thinks its the best ever .. or something ...gotta love those PHBs!)

Hi,
Thanks for the advice, I was originally planning to use HTML::Template and MySQL and build like a normal web app as you describe. You make some good points about the code that generates static pages and I'll probably stay away from that option now.

Let me know if you do go for Perl with your project and i'll share my ideas and explain the kind of system I end up with.

Cheers,
Tom

Essentially you are going to maintain a directory or index. A database would store the various aspects of teh category information and so on, be searchable in a variety of ways, but ultimately gives LINKS to the documents themselves.

You could add to the directory system a hosting management portion which allows the completed pages to be uploaded, placed in the hosting directories and have the links added to the database.

Think of the problem as a document management problem. The Google directory system only maintains a set of classifications, descriptions and links - all displayed dynamically - it does not maintain or manage any of the actual documents.

Good luck! Sounds like a fun project - but I don't think it is quite as daunting as you think.

...john

I am not sure that Mason will be mnuch help? Aren't the pages likely to be generated by individuals or groups on campus?

It depends on exactly what the OP is really trying to do.

Imagine a database schema storing links, classifications, descriptions, etc. The actual pages might be elsewhere.

Then two sets of web pages, one that allows the "individuals or groups on campus" to add/edit the information in the database, and another set that search the database and construct the pages for the end user.

To amplify a bit on what Ryszard said you need to look at what you are trying to do with a project management type of methodology. The steps you need to take are:

1. Gather requirements
2. Measure what you have
3. Formulate a transisition plan
4. Get signoff from the stakeholders
5. Fine tune the plan based on feed back from step 4
6. Iterate #4 & #5 as needed
7. Set a schedule with milestones
8. Execute the plan. Changes to the plan at this point need to be re-negotiated with a setting of expectations (see #4 and #5) as needed.
9. DOCUMENT EVERYTHING
10. Wrap up the project with a "what went well and what went wrong" session

Now: implementation wise the kinds of technologies you should be looking at would include some sort of spider that looks at meta tags within the pages to build a database of those tags indexed against titles and authors. The meta tags and how they are used would be part of the standards you establish and those pages that don't conform to the standards don't get indexed. The consequence of not being indexed should be the "management's" call and part of the expectation setting that needs to take place.

Generating the index page (or pages) can be done several different ways. Run a job once a day that spiders the site and compiles the indices. That much stays pretty constant. The part that is subject to implementation preference is wheather you generate a static HTML page as a result of the spidering OR use HTML::Mason, PHP, or other dynamic web page technology is in small part a case of how big a load you expect the web server to have to deal with, how "hefty" the machine is, personal preference and possibly political considerations.

When I say political considerations I worked in a shop for a while that had "Core Engineering Standards" that in a draconian fashion dictated what technologies were approved for use on company machines and you were not allowed to even suggest a technology not in the CES document. Something else to consider before making big plans.

Humph... wish I wasn't under NDA or I'd give you the link of the site I did all this work for... :-)

Go back and read the question again. google directory is not the same as google search.

And if the question was "How can I rewrite Google?" the answer would be "just buy it."

Thanks for all that, very useful to see the stages planned out like that. I will take it all into consideration.

Cheers,
Tom

directory.google.com is based on dmoz.org - the Open Directory Project. Unfortunately, the source to their engine is not available (but all the content is).

A quick search for dmoz on freshmeat.net showed several compatible projects, including one in perl.

I've never used any of them, so I can't vouch for them, but at least it's a pointer in the right direction.

You can also check the ODP Forums for ideas

This is assuming that you don't actively want to code this, in order to further your skills. If so, then go ahead.

Dave

A massive flamewar beneath your chosen depth has not been shown here

One of the commercial ones with which i have had good experiences so far is hyperseek.. Does a good job, it's in perl, templateble and it goes for $499, which isn't that much at all.. Might be interesting.

Link: Hyperseek

Also see Links which is $150

Finally, Websight is free, but uses PHP.

I'd say, that this is an ideal project for an XML-centric approach.

You obtain the data you need from your database and "export" the data in XML-format, which you then transform into HTML through XSLT (which can be done server-side or client side, most of the browsers now support XSLT).

No need to use a templating system as XSLT will take care of that.

The downside is of course that you need to learn XML and XSLT!

CountZero

"If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

Actually I've made a couple of these, a relatively simple one which I added to a mod_perl app and one now which is a bit bigger but unfortunately am forced to do in PHP with a preset structure - acckkk! as bill the cat would say. Also PHP is evil. Thank you.

If you use mod_perl you are probably doing it for speed. But this may be overkill and you will need to use strict and get used to watching the apache error_log. Getting into Apache is useful though, for example you could use the mod_rewrite apache module to perhaps make it look like the pages are indeed static.

Perhaps it would be useful to first get realistic page mockups done of each representative "screen" in html or as a bitmap image so you can get the full feature list through discussions with your client. Then as far as technology goes why not try and abstract out the storage mechanism and focus on the logical (algorithmic) design. It could be useful also to make a "web service" (mod_perl might be good..) which answers queries about the directory because then the directory could be more useable for other things in the future (i.e. maybe you would publish new items in an rdf feed). Of course you may not want to get into this yet but you can plan for it.

Sure mysql is probably a great idea but I wouldn't spend too much time worrying about the storage method now, instead you need to know the whole feature set and also work out the logical design you will use for representing your data. I would recommend making every item an object that for example knows its url, description, creator, parent, category, etc. Recently I had good luck with Mysql and Class::DBI in another project, you might enjoy that and it makes it relatively easy to use an OO interface to your database.

Unless you feel you just have to recreate it, I thought some of these links might help. AFAIK, each of these is written in Perl. Also, there'll be enough coding to go around in implementing anything like this into a web site, so maybe it's a good thing to use something like these since they're complete (more or less)...

• Links, shareware, free to non-commercial use, Here.

• Many others are listed here. However, look closely and make careful choices.

• Some of these have free versions that often have limited functionality (which may be all you need), and provide paid upgrades. Here's an example. (I don't know this product, just an example)

• Open source things like this at sourceforge (do your own testing.)