Your situation depends on two things. Firstly the relationship between the website owners and you. Secondly, what you do with the information.

If you are scraping the website of a competitor, someone who derives an income stream from the information or someone who has already paid someone else for the content on their website then you probably won't get very far. It is also the case that if you are re-selling the information that you get or not attributing the source of the information then you would be in trouble (breech of copyright, passing off etc.).

If, on the other hand, you are crawling a website that is otherwise in the public domain (e.g. government websites) then it may be worth getting in touch with the website owners and talking to them about it. Content owners are trying harder to provide machine to machine services such as web services, RSS feeds etc. A small licence fee later and you could end up with a web service feed rather than a web site scrape.

As a general note from a technical point of view - be nice when you are scraping/indexing.

• Index a site slowly - leaving a second or two between requests will take the pressure of the website (there's nothing like a DOS attack to annoy people). Varying the spacing in between calls also stops the requests clumping together in the website logs and drawing attention to your spider.
• Set the Agent header in your requests to include a contact e-mail so that the website admin can get in touch with you.
• Index a site only when necessary. The chances are that most of the site is staitic with only a small amount of info changing. The guiding principal is Less is more.

In terms of your relationship with your boss. You need to document your concerns and the approach that you are taking. Remember that if you can demonstrate that they knew what was happening then they get it in the neck and not you.

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.