By pull apart I mean extract data from. This program is going to help me monitor pricing on websites that my employer buys products from very frequently. We have a bandwidth shortage so we need to precache the pricing information from the pages. I am looking at a long list of pages here and I need to get the information from an entire section of every site. I have spent a few days creating some really general scripts that work with differant site configurations, and they work alright but I am still spending too much time formulating regular expressions. These sites usually are entirly out of a database with no custom code whatsoever.
-Douglas | [reply] |
If this is a company that you do large volumes of orders from have you talked to them to see if they will provide a csv dump of their product list? Scraping data like this off an order site just seems like a huge job with little payoff.
-Waswas
| [reply] |
That was absolutly the first thing I did. This DB is for rare chemicals and most of the people that I talked to (I did manage to get a couple freebies) were totally unprepared for my request.
-Douglas
| [reply] |