anything in XS an optimisation? In which case maybe we should revert to a pure perl implementation of sort, map etc.?

Are you trying to assert that sort and map are implemented via XS? My objection is about XS code, not that the Perl source code is written in C.¹

Mind you, there is no guarentee that the perl version will be bug-free.

There are few true guarantees in life. I'm not sure why you felt I was trying to offer one or assert the existence of one. But I'd be happy to bet you that this bug will go away if you follow your own instructions on how to remove the XS code. My assertion is that XS code is more likely to be buggy than Perl code. Actually, I find it to be far more likely to be buggy.

Nor is there any evidence that shows this to be a bug that results from an optimisation, unless you consider writing anything in XS an optimisation?

No, I don't consider writing anything in XS to be an optimization. But you tell me, why would one have the same functionality written in both Perl and XS? Why is the XS code present in this case? I'm pretty sure the primary (probably only) reason for this XS code is an attempt to increase "performance".

Examine what is said.

                - tye

¹Perhaps you would like to implement "reduce" as a Perl keyword and then compare the resulting code to the current XS code in order to appreciate that the two ways of implementing Perl functionality in C are quite different. Then try to get your code included into Perl and compare that to getting your XS code into CPAN.

Forgive me Tye,

Examine what is said.

I really thought that I had, and was responding to what you had said. I'll admit that I took a few leaps forward from your exact words to what I considered to be their logical conclusion. Namely:

• That code should not be optimised as it it harder to get right and that using C (via XS) to code something that can be coded in Perl is an undesirable thing to do for that reason.

If this is a misinterpretation of your words or intent, I apologise.

I'll readily admit to being anything but an expert in either the perl sources or XS, but when I looked at the source for sort and attempted to compare it with the source for reduce, I found the following two routines

Read more... (2 kB)

The former is only that part of sort that deals with the setting of $a & $b, calling the comparator and retrieving & returning the result. Ie. That bit that is directly analogous to the reduce code, and to my eyes they are very similar. The latter example is more complicated, but then the former is only 28 lines from nearly 2000 that implement the built in sort. Many of the macros used seem (to my inexperienced eyes) to be very similar.

So, whilst I wasn't asserting that they were the same, I was suggesting that the level of difficulty in developing bug-free routines in either was similar. To my knowledge List::Util is fairly immature code relative to sort--even the latest implementation of which are derivative of prior art and show the hands of many people in its evolution--so I find it unsurprising that there are still bugs. Only through use will these be discovered and corrected.

But you tell me, why would one have the same functionality written in both Perl and XS?

In a way, that was my point exactly. sort could be provided as a pure perl implementation. The reason it isn't is mostly performance. The frequency of use, and therefore the frequency (as well as the size) of the performance gains that accrue from implementing it in C (or XS which resolves to C) are such that the extra work involved is deemed worth it.

I contend that reduce is a similarly, generically useful routine that it too will benefit the many by the extra effort expended by the few to make it run as efficiently as is practical. That List::Util has been adopted into the core whilst still remaining an extension rather than being moved into the perl source proper indicates that perhaps others hold similar views. It may well mean that the move was premature as there are still bugs, but as I said previously, there are still bugs in the perl source proper (eg.substr), so it's really a matter of opinion.

---

Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller
If I understand your problem, I can solve it! Of course, the same can be said for you.

First, I don't think that the primary reason that map and sort are implemented in C is because of performance. They were chosen to be added to the language and so they were implemented as part of the language (which is implemented primarily in C).

If you feel that "reduce" is useful enough and the speed difference is important enough, then by all means advocate a patch to the Perl source so that the level of review and maintenance of the code will rise to match the high level of difficulty of writing C code that manipulates Perl structures. Most XS code shares this difficulty but does not share most of the review (of both design and code), testing, maintenance, or even expertise of the Perl source code.

XS code also has a different environment that makes it even more likely to have bugs than Perl source code. XS authors routinely make simplifying assumptions that Perl developers do not (even in the case of a single person playing those two different roles).

I don't find the two subroutines you quoted to be very similar.

As to your restatement of my thesis:

That code should not be optimised as it it harder to get right and that using C (via XS) to code something that can be coded in Perl is an undesirable thing to do for that reason.

It isn't too far off. Might I suggest that in future, when you respond to a reinterpretation of what someone has written (which, really, is always what we do), for each point in your response, compare the point against the original statement. I find that this often helps me to realize when I've over- or mis-interpreted what someone has written (or simply that I'm not sure what they meant in relation to that specific point). Just yesterday and today I've deleted several draft paragraphs as a result of this self-check.

But it is a bit off:

code should not be optimised

I specifically mentioned "enthusiastic" optimization. You appear to have interpreted several of my points as "black-and-white" that I did not (IMHO) express that way. There are certainly good reason to optimize. All too often I see great effort being put toward "optimization" that I consider simply foolish.

This specific case of optimization falls perhaps a bit short of the "foolish" label, but goes beyond what I consider sensible.

using C (via XS) to code something that can be coded in Perl is an undesirable thing to do for [optimization]

That is close. But just a few weeks back I helped someone implement C code to speed up a tree search and just last week I posted Re: checking for overlap (index) which proposes "moving much of the search code into [C]".¹ I specifically mentioned Inline::C and not XS, but I don't approve of all Inline::C code nor condemn all XS code.

What I condemn is writing C code that manipulates Perl structures (other than as part of Perl, where the practice is required and properly dealt with). I've seen XS code written by some of the best XS coders around (near as I can tell) and I'm usually impressed with how bad it is.

I've never seen anyone asking for help with bugs in C code written for Perl extensions when that code avoided dealing with Perl structures. Inline::C helps you avoid dealing with Perl structures in your C code more than XS does. But the important point is to avoid it, whichever tool you use.

So, optimizing something that works perfectly well when you have no profiling saying that it takes too long to run by trying to only eliminate the opcode dispatch work by reimplementing it in a particularly buggy form of C code that you hope is mostly the same as the code that gets run in the original case except without all of the review/testing/maintenance infrastructure ... No, it doesn't seem sensible to me.

Maybe the XS code is hundreds of times faster and the huge risk increase is worthwhile. I don't know. That was part of why I had "</soapbox>" and a smiley in the original node.

                - tye

¹If I were vain enough to assume that you had noticed these, then I'd suggest you "examine who speaks" ;)