Just remembered something else that might be useful..

When it comes time to rate projects in terms of qualities X1..Xn, don't ask people to look at a single project and assign numbers for each axis. The numbers you get will vary from person to person, according to context. Most people would say that a 50' frog is big, for example. But those same people would also say that the ocean is big, and that the entire known universe is big. If you ask people rate the 'bigness' of those three items, the numbers you get will vary based on the order in which they're presented.

OTOH, pretty much everyone will agree that the ocean is bigger than a 50' frog, but smaller than the known universe. In other words, people's perception of relative X-ness tends to be more reliable than their absolute rating.

So instead of rating projects in isolation, give people two projects and ask them to compare them in terms of qualities X1..Xn. For each pair, you'll get a set of n more/same/less ratings that place the two projects relative to each other. Then once you collect enough ratings, you can distribute items along each axis by sorting them topologically.

It is possible that the ratings you collect will be inconsistent, especially if you have more than one person ranking projects. One person may say P1 is bigger than P2, another may say P2 is bigger than P3, and a third may say P3 is bigger than P1. Topologically, that's a loop, and will screw you up if you try to convert ratings to absolute positions along the scale.

You solve that problem by collecting more data, and making the relative values fuzzy. If 60% of people say P1 is bigger than P2, you say that 60% of P1 is bigger than P2, but 40% is smaller. If 30% of people say P2 is bigger than P3, P2 outs P3 by a 30/70 margin. If only 10% of people say that P3 is bigger than P1, it means that 10% of P3 is still bigger than some part of P1: 

    P1           ----------
    P2     ----------
    P3  ----------

[download]

Yes, that's a contrived example.. I'm doing text-graphs, here. ;-) The same idea applies for less typeable values, though.

If you really want to obsess about it, you can generate a range and distribution function for each item along each axis, which is pretty much the definition of a fuzzy point. Then you can select a region from the space, and rate projects in terms of how much each one falls inside that region.

Hope I haven't stepped over the line into hopeless incomprehensibility.

In reply to Re: Re: Comparitive Scoring System by mstone
in thread Comparitive Scoring System by gwhite

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.