You should include a link to one of the BBC forum/comment pages for anyone to peruse should they want to help you. You should first check the site’s Terms of Service. Automated queries are proscribed on most big sites and most of the better hackers here won’t help break such.

Many comment boards on news sites do Ajax updating now. If this is one of those cases then WWW::Selenium or WWW::Mechanize::Firefox (as suggested already) would probably do it. Plain WWW::Mechanize might work too but would be much more intrusive on the server and more obvious. I know Selenium has watcher/event functions/JS access so you can wait for comments to appear and then deal with them.

You will find bias. It’s not even a question, is it? And therein lies the real rub. You’ll find it and present it and those who are biased won’t see it and everyone else (the minority that can divorce preference/proclivity/upbringing from right/wrong/logic) will say… well, DER–P! The insanity of the whole thing boggles, really. But it is a marketplace, mr_market, in’it? I think this is a fun project and wish you luck and will even help you if the site’s ToS allow and you post follow-ups here but maybe starting your own political/news/whatever forum might be a better use of your frustration/time/potentially-righteous-anger.

(Update: added possessive s.)

In reply to Re: Web Have Your Say diff log by Your Mother
in thread Web Have Your Say diff log by mr_market

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.