Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot

Re: scraping from HTTP page to MySql table

by z3d (Scribe)
on May 03, 2004 at 12:55 UTC ( #349990=note: print w/replies, xml ) Need Help??

in reply to scraping from HTTP page to MySql table

Like the posts before me, I won't offer code, only recommendations and insight. I would start by warning you - unless you run the website you are scraping, or have an existing relationship with the owners, you may want to think twice about a direct scraping every two minutes. Not everyone appreciates having their website hit repeatedly and consistantly to scrape data.

In addition to the modules already mentioned, I'd also recommend reading through past articles. I know that both and TPJ have run articles about exactly this, in the last few months (so it might still be found on their front page, not sure).

"I have never written bad code. There are merely unanticipated features."
  • Comment on Re: scraping from HTTP page to MySql table

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://349990]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (1)
As of 2023-03-24 07:21 GMT
Find Nodes?
    Voting Booth?
    Which type of climate do you prefer to live in?

    Results (60 votes). Check out past polls.