in reply to Re: Seeking input on pattern generation.
in thread Seeking input on pattern generation.

By pull apart I mean extract data from. This program is going to help me monitor pricing on websites that my employer buys products from very frequently. We have a bandwidth shortage so we need to precache the pricing information from the pages. I am looking at a long list of pages here and I need to get the information from an entire section of every site. I have spent a few days creating some really general scripts that work with differant site configurations, and they work alright but I am still spending too much time formulating regular expressions. These sites usually are entirly out of a database with no custom code whatsoever.

-Douglas
  • Comment on Re: Re: Seeking input on pattern generation.

Replies are listed 'Best First'.
Re: Re: Re: Seeking input on pattern generation.
by waswas-fng (Curate) on Jul 22, 2003 at 23:46 UTC
    If this is a company that you do large volumes of orders from have you talked to them to see if they will provide a csv dump of their product list? Scraping data like this off an order site just seems like a huge job with little payoff.

    -Waswas
      That was absolutly the first thing I did. This DB is for rare chemicals and most of the people that I talked to (I did manage to get a couple freebies) were totally unprepared for my request.
      -Douglas