in reply to How can I get all URL's of a domain and get text (scraping)?

are people still doing scraping manualy? Try an automated solution
https://www.kimonolabs.com/

What you get for free?
*picking visually the fields you are interested in
*tweaking the query with CSS selectors
*an API and export captured data in a variety of formats
*schedule when the crawling should run
  • Comment on Re: How can I get all URL's of a domain and get text (scraping)?

Replies are listed 'Best First'.
Re^2: How can I get all URL's of a domain and get text (scraping)?
by Anonymous Monk on Sep 18, 2015 at 22:45 UTC

    are people still doing scraping manualy? Try an automated solution

    Right, sure, "automated" in that its a program you write through visual interface, that runs on schedule ... like any program you might write using your fingers