You should include a link to one of the BBC forum/comment pages for anyone to peruse should they want to help you. You should first check the site’s Terms of Service. Automated queries are proscribed on most big sites and most of the better hackers here won’t help break such.

Many comment boards on news sites do Ajax updating now. If this is one of those cases then WWW::Selenium or WWW::Mechanize::Firefox (as suggested already) would probably do it. Plain WWW::Mechanize might work too but would be much more intrusive on the server and more obvious. I know Selenium has watcher/event functions/JS access so you can wait for comments to appear and then deal with them.

You will find bias. It’s not even a question, is it? And therein lies the real rub. You’ll find it and present it and those who are biased won’t see it and everyone else (the minority that can divorce preference/proclivity/upbringing from right/wrong/logic) will say… well, DER–P! The insanity of the whole thing boggles, really. But it is a marketplace, mr_market, in’it? I think this is a fun project and wish you luck and will even help you if the site’s ToS allow and you post follow-ups here but maybe starting your own political/news/whatever forum might be a better use of your frustration/time/potentially-righteous-anger.

(Update: added possessive s.)