Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Re: Perl Search Applicance

by PyroX (Pilgrim)
on Jun 20, 2002 at 16:24 UTC ( #176078=note: print w/replies, xml ) Need Help??


in reply to Perl Search Applicance

All the input has been great. Plus I have the urge to actually write the thing now. I am thinking about going ahead with my idea for a "overlord" of types, I would like to be able to logon, see what pages are being looked at, start/stop any of the bots, and change discard links I dont want them to follow that they are coming up on.

I am thinkin about a 4 process system, the overlord, and 3 bots, I want to make them slow and smart, I don't really care if it takes them a day to get through 1000 pages, or even less.

My main concern is the response time for the searches.

What do you (all) recommend for the database? Remember I will not spend a bunch of $$$ for anything. I know how horrible MySQL is with over 500,000 of anything (ie rows in a table). I need more guidance with that portion.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://176078]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (5)
As of 2022-12-05 23:46 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?