Metrics tracking Perl 6 development

I write #perl6 summaries. An AM recently said "This is nice. Also can you give a percentage progress of how much of rakudo matches the spec as it exists today.".

I don't think we can (reasonably easily) do exactly as the AM requested. I am also concerned about encouraging the impression Perl 6 is not feature complete (for some users and use cases it is now sufficiently complete).

However, I would like to publish a metric that helps the AM and interested parties track Perl 6 development. I would appreciate constructive feedback to my response and thoughts below.

TL;DR As a metric to help track Perl 6 development, I favor test pass rates such as spectest passes over time or "64 modules passing 100% of their tests when compiled using the 2012.08 Rakudo compiler". Do you like these?

Any metric will almost certainly have to be based on data that's already being collected. I could see trying one or more of the following:

Tracking completion of very high level features of compiler implementations. What do you think of Feature completion pie charts? This is drawn from the same data used for the compiler comparison matrix.
We could update the pie chart weekly.
This data currently tracks 143 very high level compiler "features", ones that have been mostly implemented in five years or so (Rakudo) or 2 years or so (Niecza) for an average implementation rate in the order of 20 - 40 per year.
Some issues:
- 143 entries is a small list. The rate of change is slow. Many weeks it wouldn't change. It will be non-trivial to expand this list in a way that significantly increases its utility for tracking Perl 6 development (as against comparison among compilers).
- Some important "features" are not included in this compiler feature list. Consider the (awesome!) debugger that was just added to the latest Rakudo Star bundle. The compiler features list didn't include a debugger. (As it happens, the spec didn't say much either -- "needs further specification", and the Rakudo roadmap didn't even mention it.)
  I suspect the fact this feature list doesn't include the debugger reflects weakness for tracking what the AM cares about.
- There are non-essential items in the compiler features list. For example, junctions, hypers, and races are spec'd to optionally autothread (automatically provide transparent concurrency). Rakudo implements junctions and hypers but does not autothread.
  Do we count Rakudo as having the junction and hyper features? If we do, folk are likely to complain that this is misleading because it doesn't yet do autothreading. Do we include the optional autothreading in the "features not yet implemented" count? If we do, folk are likely to complain that Rakudo isn't ready because it's not "feature complete".
Tracking test passing percentages. Consider the Perl 6 test suite and the related perl6 pass rates.
This data generally changes at least a little from week to week.
I think this is about the best data we have that can quickly convey progress but publishing it has tended to elicit criticism ("Test count is a silly metric.").
In the past flussence has generated charts of spectest passes over time. He gave up on it recently, perhaps due to it closing in on 100%, or maybe due to the negative responses we've gotten from publishing such data, but I imagine it would be fairly simple to restart this.
There are module smoketesting results. We could list how many modules in the Perl 6 ecosystem are passing 100% of their tests (64 as of 2012-09-05 tested against the August Rakudo compiler.)
A couple days ago mst uploaded Perl 6 to cpan. That's going to bring in a lot of interesting platform test data. Maybe we track that?
Tracking compiler roadmap edits. The Rakudo roadmap currently contains 60 items.
The roadmap is probably even less suitable as the subject of a weekly update than either of the previous two items. But I thought it worthy of mention here because it is a very important project driver.

A few weeks ago (on August 20th) I wrote How things are going compared to rakudo roadmap using jan 1 2012 as baseline.
Since I wrote the above gist there have been two roadmap changes. Removing (marking as done) the one star (very easy) task "sigilless variables" and removing the five star (very difficult) task "design, implement and switch to QAST".
(While we're here, let's summarize the remaining roadmap tasks. There is just one task marked as top priority: "basic Perl 5 interop (use, eval, etc.)". Interestingly it is not marked as being difficult. Then come 38 medium priority tasks (only two marked difficult); 17 low priority tasks (only one marked difficult); and one task of unspecified priority and difficulty ("Correct type smiley support (:U, :D, :T, etc.)").)

Do you particularly like or dislike any of the above? Any other ideas?

Edited to correct statements of roadmap task difficulty.

Comment on Metrics tracking Perl 6 development

Replies are listed 'Best First'.
Re: Metrics tracking Perl 6 development by moritz (Cardinal) on Sep 09, 2012 at 20:02 UTC
The only metric that matters is "can you do with it what you want to do?". Readiness cannot be meaningfully projected onto a simple number, it's always a question of how your needs and the compiler and ecosystem fulfill those needs. For example most people say that Perl 5 is production ready, but it's still pretty much useless for number crunching/high performance computing -- so it's not ready for this particular use case. So, what readiness number would you give Perl 5, even if it's not ready for every possible use case? Tracking implemented features is useful, because people want to know if they can use a particular feature. But doing that as a percentage of a pretty much arbitrary goal isn't very useful, because the current usefulness isn't really a function of future plans. The same holds true for the tests: if we add or remove tests without changing the compiler, the readiness metric changes. In the end the best thing to do is probably to build cool stuff with Perl 6 (or improve the compiler) and talk about it, instead of trying to come up with complicated (and yet still not very good) ways to measure progress. Perl 6 - the future is here, just unevenly distributed	[reply]
Re^2: Metrics tracking Perl 6 development by Anonymous Monk on Sep 10, 2012 at 08:08 UTC
Something is seriously wrong with you if you say Perl 5 is not ready for production use. By your definition a rocket is not production ready to launch into space because it cannot run on roads. This argument is absurd, and you diverting from the topic. Even if you by any means of semantic play deduce Perl 5 is not ready for production use. Can you please answer the inevitable question Is Perl 6 at least as much production ready as much as Perl 5 is? That question captures what people mean when they ask for production readiness of Perl 6. And there are no complicated measurements for productions readiness, some how only Perl 6 is having all these complications, every other piece of compiler in this world slaps '1.0.0' at a point of time with sufficiently acceptable conditions of quality, feature completeness, libraries and documentation and ships it. If you are not there yet, then you are not there yet. Making the definition the goal ambiguous doesn't mean you achieved your goal.	[reply]
Re^3: Metrics tracking Perl 6 development by moritz (Cardinal) on Sep 10, 2012 at 13:55 UTC
Is Perl 6 at least as much production ready as much as Perl 5 is? For what use cases? You see, Perl 6 isn't meant as a successor for Perl 5 (at least not anymore; that was the original intention for Perl 6, but we've diverted from that path long ago), so it's not clear to me that Perl 6 has to excell in exactly the same spots as Perl 5. So I don't think we need to have a complete superset in productivity to declare it production ready. That question captures what people mean when they ask for production readiness of Perl 6. Citation needed. Maybe it captures your idea, but people I talk to all have different ideas of production readiness. That question captures what people mean when they ask for production readiness of Perl 6. I frankly admit that on most areas, existing Perl 6 compilers cannot compete with Perl 5 on most "production readiness" metrics. So, if that's the answer you want to hear, please have it and be happy. But there are a lot of folks who don't need the same level of stability and speed, and who are willing to give up a bit of both in exchange for a much more expressive and consistent language. And for those folks we need a more detailed answer than "yes" or "no". It's for those folks that we blog about our progress, fix bugs, add features and make monthly releases -- not for the anonymous crowd waiting for a release labeled 1.0 Perl 6 - the future is here, just unevenly distributed	[reply]
Re^2: Metrics tracking Perl 6 development by raiph (Deacon) on Sep 12, 2012 at 19:40 UTC
Thanks Moritz. That was a helpful response. I've concluded that if any of the existing metrics I discussed get mentioned on #perl6 they'll probably appear in a #perl6 summary like anything else, but I won't special case any of them. I've also concluded there's a need for something new. More on that if/when I have something to show.	[reply]