Anybody wanting to advance perl 5 seriously, should consider starting from scratch!

Its called Perl 6.

Unfortunately in relative terms, that accounts for mostly nothing. 99% of the complexity of the perl interpreter/compiler/runtime is due to its initial design.

That's why you reduce & refactor.

30 something years ago I was contracted by IBM to work a maintenance gig on DB2. It was at that point written in COBOL of which I had minimal experience. I did a 4 week course on it at college.

I (typically) went into the APAR database and picked the longest outstanding bugs to tackle. One example was a sev.4 that had been outstanding for 4 years. I spent 3 days trying to understand the bug report in the context of the code; and two more badgering a user to reproduce the problem.

Once I understood the problem I tracked the bug to a 2500 line procedure that, upon inspection, although I understood what it was meant to do, I couldn't, from reading the code, work out how it did it!

So, I threw away the entire body of the procedure, retaining only the inputs and outputs and set about rewriting it to perform the task it was documented as performing. The result was the reduction of a 783 (those who know binary will understand why the figure sticks in my brain), chunk of code to around 20 lines.

1. It fixed the problem.
2. It functioned correctly. Ie. It passed all existing tests plus a bunch I added.
3. It was rejected.

   I could not identify the specific line or lines of code that cured the APAR.

It was the only contract I arranged to leave early -- I bought my way out of it.

Two conclusions:

1. Set-in-stone rules and procedures are the single biggest barrier to progress.

   p5p has lots of them.

2. Any piece of code greater than 20 lines can be improved; especially if freed of the constraints of history.

   Some of Perl's functions are 1000s of lines long.

You may be right -- you usually are -- but, given sufficient will, I'd be prepared to expend some time to trying to prove you wrong.

C'est la vie.

It was called Perl 6 and then it went insane.

A complete rewrite of the guts with the cleanup and change of a few warts and with a few additions would make sense and would rightfully be called Perl 6. A bit akin to what Topaz planed to be, but without the decision to use C++. It might even make sense, if the project started a little later, to target the java or .net runtime (think Mono).

It might even have been released about ten years ago while "the inner circle" would be busy writing the synopses of a new megalanguage with a cool codename intended to become the Perl successor sometime in the future or a little later.

Jenda
Enoch was right!
Enjoy the last years of Rome.

Anybody wanting to advance perl 5 seriously, should consider starting from scratch!

Though I too would love to see that, I doubt it is realistic. Remember, both Topaz and Ponie were attempted and abandoned. The best we can hope for is probably a slow, relentless refactoring.

Of course, the "old, huge, tangled code base problem" is not specific to the Perl 5 internals. It is a chronic problem in the software industry, even afflicting the blessed Perl Monks code base. :)

The problem isn't an infrastructure issue, however -- and speaking as one of the handful of people who've had a hand in developing the site's software: It's our own gosh-darn fault. Perlmonks is WAY more complex than when it originally launched. It does a crapload of perl evals and sql queries per page. It's vulnerable to resource hogs. Searching can cripple the database. And right now, I don't think we're gonna fix these problems any time soon. ... It's not a matter of computer resources, as much as human engineering resources.

-- Re: perlmonks too slow by nate (original co-author of the Everything Engine)

Some relevant quotes from Nobody Expects the Agile Imposition (Part VI): Architecture follow:

Have you ever played a game called Jenga? The idea behind Jenga is that you start by making a tower of blocks. Each player removes a block from somewhere in the tower, and moves it to the top of the tower. The top of the tower looks tidy, but it's very heavy and the bottom of the tower is growing more and more unstable. Eventually, someone's going to take away a block from the bottom and it'll all fall down.

I came into Perl development quite late, and I saw a very intricate, delicate interplay of ideas inside the Perl sources. It amazed me how people could create a structure so complex and so clever, but which worked so well. It was only much later that I realised that what I was seeing was not a delicate and intricate structure but the bottom end of a tower of Jenga. For example, fields in structures that ostensibly meant one thing were reused for completely unrelated purposes, the equivalent of taking blocks from the bottom and putting them on the top.

-- The Tower of Perl by Simon Cozens

The perl5 internals are a complete mess. It's like Jenga - to get the perl5 tower taller and do something new you select a block somewhere in the middle, with trepidation pull it out slowly, and then carefully balance it somewhere new, hoping the whole edifice won't collapse as a result.

-- Nicholas Clark

I remember one company that has about 120 engineers, developers of all kinds of whom 10 are still able to work on the core functionality. The other 110 are working on new stuff. We brought all the engineers into the room. We said, okay, the product manager for the first area and the lead engineer for the first area come on up here. Now select the people you need to do this work over the next month, including, of course, the core engineers. And they did and we said, okay, now leave, get out of here and start working. ... when we got to the fifth product manager and the lead engineer and they said we can't do anything. There's no core engineers left. We looked around the room and there were 60 engineers left. They were thoroughly constrained by the core piece of functionality.

If you have enough money, you rebuild your core. If you don't have enough money and the competition is breathing down your neck you shift into another market or you sell your company. Venture capitalists are into this now, buying dead companies. Design-dead software.

-- Ken Schwaber, Google tech talk on Scrum, Sep 5, 2006 (38:40)

Netscape 6.0 is finally going into its first public beta. There never was a version 5.0. The last major release, version 4.0, was released almost three years ago. Three years is an awfully long time in the Internet world. During this time, Netscape sat by, helplessly, as their market share plummeted. It's a bit smarmy of me to criticize them for waiting so long between releases. They didn't do it on purpose, now, did they? Well, yes. They did. They did it by making the single worst strategic mistake that any software company can make: They decided to rewrite the code from scratch.

It's important to remember that when you start from scratch there is absolutely no reason to believe that you are going to do a better job than you did the first time. First of all, you probably don't even have the same programming team that worked on version one, so you don't actually have "more experience". You're just going to make most of the old mistakes again, and introduce some new problems that weren't in the original version.

-- Joel Spolsky on not Rewriting

Now the two teams are in a race. The tiger team must build a new system that does everything that the old system does. Not only that, they have to keep up with the changes that are continuously being made to the old system. Management will not replace the old system until the new system can do everything that the old system does. This race can go on for a very long time. I've seen it take 10 years. And by the time it's done, the original members of the tiger team are long gone, and the current members are demanding that the new system be redesigned because it's such a mess.

-- Robert C Martin in Clean Code (p.5)

That's not to say it can't be done though. The great Netscape rewrite (ridiculed by Spolsky above) -- though a commercial disaster -- metamorphosed into an open source success story. Another example of a successful rewrite is the Perl 5 rewrite of Perl 4.

The best we can hope for is probably a slow, relentless refactoring.

Relentless I agree with; but I question "slow".

It is my (long) considered opinion that if you reduce the targets to Windows and *nix (I'd say POSIX, but that lives about a decade behind (at least) linux); then with just 10 people (They'd have to be the right people), you could refactor perl5 in one year, to be:

• Faster.
• Simpler.
• More maintainable.

The secret: Fcuk the rest; get it working on these two first.

Rational: If it can be made to work -- ie. passes the entire perl build test suite and perlbench on those two platforms, whilst having reduce the kloc by 50% and the average function size by 50%-- on those two wildly disparate platforms, then it can be made to work anywhere where there is sufficient will and bodies to tackle the task. (If there ain't; c'est la vie!)

Detail: Whilst neither kloc nor function size is directly proportional to understanding, correctness and maintainability ; the correlation is so strong, over many studies over many decades, that it would be obtuse not to recognise that simplification is inversely proportional to understanding; and understanding is directly proportional to both correctness and maintainability.

Target: Start small in terms of platforms (just two); start small in terms of functionality (just does what p5 does now); start small in terms reduction in size. I've suggested 50% but I believe 70% is (quite easily) achievable.

If you reduce the current code by 50%; twice as many people stand a chance of understanding it.

Then you emulate the pugs model: everyone gets a commit bit; (a majority of) 3 people have to agree, to rescind a commit.

The guiding logic

Nothing substantial -- syntax or semantic -- changes until a 50% reduction (compiled kloc) occurs. Then you invite both requests-for-change, and patches.

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Of course, the "old, huge, tangled code base problem" is not specific to the Perl 5 internals. It is a chronic problem in the software industry

Yes, but perl 5 is an extreme case.

In any case, IIRC topaz was not a fiasco but a prove of concept that derived into the Perl 6/Parrot project. Ponie intention was to bridge XS and parrot, something dammed from the start because XS is just a way to access the perl 5 internals directly. A high level API hidden the implementation details is completely missing.

Corollary: anybody wanting to advance perl 5 seriously should forget about XS compatibility. It is a too heavy to carry stone.