Forgive me Tye,

Examine what is said.

I really thought that I had, and was responding to what you had said. I'll admit that I took a few leaps forward from your exact words to what I considered to be their logical conclusion. Namely:

• That code should not be optimised as it it harder to get right and that using C (via XS) to code something that can be coded in Perl is an undesirable thing to do for that reason.

If this is a misinterpretation of your words or intent, I apologise.

I'll readily admit to being anything but an expert in either the perl sources or XS, but when I looked at the source for sort and attempted to compare it with the source for reduce, I found the following two routines

The former is only that part of sort that deals with the setting of $a & $b, calling the comparator and retrieving & returning the result. Ie. That bit that is directly analogous to the reduce code, and to my eyes they are very similar. The latter example is more complicated, but then the former is only 28 lines from nearly 2000 that implement the built in sort. Many of the macros used seem (to my inexperienced eyes) to be very similar.

So, whilst I wasn't asserting that they were the same, I was suggesting that the level of difficulty in developing bug-free routines in either was similar. To my knowledge List::Util is fairly immature code relative to sort--even the latest implementation of which are derivative of prior art and show the hands of many people in its evolution--so I find it unsurprising that there are still bugs. Only through use will these be discovered and corrected.

But you tell me, why would one have the same functionality written in both Perl and XS?

In a way, that was my point exactly. sort could be provided as a pure perl implementation. The reason it isn't is mostly performance. The frequency of use, and therefore the frequency (as well as the size) of the performance gains that accrue from implementing it in C (or XS which resolves to C) are such that the extra work involved is deemed worth it.

I contend that reduce is a similarly, generically useful routine that it too will benefit the many by the extra effort expended by the few to make it run as efficiently as is practical. That List::Util has been adopted into the core whilst still remaining an extension rather than being moved into the perl source proper indicates that perhaps others hold similar views. It may well mean that the move was premature as there are still bugs, but as I said previously, there are still bugs in the perl source proper (eg.substr), so it's really a matter of opinion.

First, I don't think that the primary reason that map and sort are implemented in C is because of performance. They were chosen to be added to the language and so they were implemented as part of the language (which is implemented primarily in C).

If you feel that "reduce" is useful enough and the speed difference is important enough, then by all means advocate a patch to the Perl source so that the level of review and maintenance of the code will rise to match the high level of difficulty of writing C code that manipulates Perl structures. Most XS code shares this difficulty but does not share most of the review (of both design and code), testing, maintenance, or even expertise of the Perl source code.

XS code also has a different environment that makes it even more likely to have bugs than Perl source code. XS authors routinely make simplifying assumptions that Perl developers do not (even in the case of a single person playing those two different roles).

I don't find the two subroutines you quoted to be very similar.

As to your restatement of my thesis:

That code should not be optimised as it it harder to get right and that using C (via XS) to code something that can be coded in Perl is an undesirable thing to do for that reason.

It isn't too far off. Might I suggest that in future, when you respond to a reinterpretation of what someone has written (which, really, is always what we do), for each point in your response, compare the point against the original statement. I find that this often helps me to realize when I've over- or mis-interpreted what someone has written (or simply that I'm not sure what they meant in relation to that specific point). Just yesterday and today I've deleted several draft paragraphs as a result of this self-check.

But it is a bit off:

code should not be optimised

I specifically mentioned "enthusiastic" optimization. You appear to have interpreted several of my points as "black-and-white" that I did not (IMHO) express that way. There are certainly good reason to optimize. All too often I see great effort being put toward "optimization" that I consider simply foolish.

This specific case of optimization falls perhaps a bit short of the "foolish" label, but goes beyond what I consider sensible.

using C (via XS) to code something that can be coded in Perl is an undesirable thing to do for [optimization]

That is close. But just a few weeks back I helped someone implement C code to speed up a tree search and just last week I posted Re: checking for overlap (index) which proposes "moving much of the search code into [C]".¹ I specifically mentioned Inline::C and not XS, but I don't approve of all Inline::C code nor condemn all XS code.

What I condemn is writing C code that manipulates Perl structures (other than as part of Perl, where the practice is required and properly dealt with). I've seen XS code written by some of the best XS coders around (near as I can tell) and I'm usually impressed with how bad it is.

I've never seen anyone asking for help with bugs in C code written for Perl extensions when that code avoided dealing with Perl structures. Inline::C helps you avoid dealing with Perl structures in your C code more than XS does. But the important point is to avoid it, whichever tool you use.

So, optimizing something that works perfectly well when you have no profiling saying that it takes too long to run by trying to only eliminate the opcode dispatch work by reimplementing it in a particularly buggy form of C code that you hope is mostly the same as the code that gets run in the original case except without all of the review/testing/maintenance infrastructure ... No, it doesn't seem sensible to me.

Maybe the XS code is hundreds of times faster and the huge risk increase is worthwhile. I don't know. That was part of why I had "</soapbox>" and a smiley in the original node.

                - tye

¹If I were vain enough to assume that you had noticed these, then I'd suggest you "examine who speaks" ;)

Well, it seems that I wasn't so far away in interpreting your intent.

To me, writing this type of extension in XS seems like a reasonable way of adding new functionality to the core without requiring adding new keywords to the tokeniser and messing with the parser etc. with all the inherent risks that entails.

Never haven written a line of XS, I'm not really in a position to judge, but given the credentials of the author of List::Util, I'd have to assume that he knows what he is doing.

I don't quite follow your point regarding review/testing/maintainance? Isn't any code that get distributed as a part of the core subject to the same scrutiny and mainatainance, be it C/Perl/or XS?

In terms of the performance gain of doing so, in this particular instance a crude benchmark shows that the XS version is approximately 3 times faster than the pure perl version.

P:\test>perl -MList::Util=reduce -MBenchmark=timethis 
-le" timethis -1, q[ our $t=reduce{ $a + $b } 1 .. 10000; ]; print $t"
timethis for 1:  1 wallclock secs 
( 1.04 usr +  0.00 sys =  1.04 CPU) @ 35.54/s (n=37)

50005000

P:\test>ren \lib\auto\List\Util\Util.dll *.lld

P:\test>perl -MList::Util=reduce -MBenchmark=timethis 
-le" timethis -1, q[ our $t=reduce{ $a + $b } 1 .. 10000; ]; print $t"

timethis for 1:  1 wallclock secs 
( 1.19 usr +  0.00 sys =  1.19 CPU) @ 10.07/s (n=12)

50005000
[download]

I guess it depends where your coming from, what you have to achieve and how long you have to achieve it, as to whether this level of performance gain is worth the risks.

So, optimizing something that works perfectly well when you have no profiling saying that it takes too long to run...

I'm not sure what you base that on...maybe prior knowledge to which I'm not party. None the less, the any piece of re-usable utility code like this, can and will be used in situations where performance is a priority. It would be impossible to profile every possible use in advance. So, if that code can be made to work 3x faster without compromising its correctess (a criteria that we now know has yet to be achieved under some circumstances), then every future use of it benefits.

Finally, I read everything that you (too infrequently:) post. I realise that you are not cursed with excessive time on your hands as I am. As always, I read what you write, and try to apply my own brand of logic to your assertions and make up my own mind. On some things I agree with you, others I don't. And some, like this, I don't yet have the experience of the subject matter to draw my own conclusions. Where my instincts go against your conclusions, I question, so as to get a clearer picture to aid me in making up my own mind on the subject.

Taking nothing as read is a curse I've suffered from since school, but I wouldn't change it. Reaching my own conclusions has stood me in pretty good stead so far.

---

Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller
If I understand your problem, I can solve it! Of course, the same can be said for you.