I think your meditation is misnamed then. The problem is not using a module: the problem is using the wrong module simply because you're familiar with it. For example, Parse::Yapp makes a different kind of parser (LALR or something like that) that might have solved your issues lickity split.

That could be true as well, merlyn. Recursive descent (the methodology) is the right way to do this sort of thing, but what TheDamian's module does isn't exactly classic RD. It is hard to do this in P::RD where it might be better done in a YACC-style parser because of the left recursion issue.

I was as frustrated with using the module itself as I was with the slowness of the program. Part of the reason I rewrote it was so that I could build my own trace functions and examine my own variables and stacks. I felt that understanding TheDamian's code would have taken me as long as the approach I took, and might not have given me the same results. That's the main reason I focused on the use of modules, and it's a broader concern than just with P::RD. As you say, any module and / or approach has its plusses and minuses and these must be evaluated for each problem one approaches.

Recursive descent (the methodology) is the right way to do this sort of thing

No, this is apparently the problem. RD is great for some things, but not for this. There's a reason that gcc is not RD, but LALR. Again, I'm a bear of very little brane about this, but when TheDamian explained it to me, it made sense at the time. It doesn't matter how fast you speed it up: something with 22 levels of precedence should be parsed with LALR (if it can), not RD.

-- Randal L. Schwartz, Perl hacker
Be sure to read my standard disclaimer if this is a reply.

-Scott

Absolutely true. The module worked properly and did solve the problem. I redid it because they want to be able to iterate parameter changes quickly and examine boundary conditions in their simulations.

As another advice, as long as you're deep in parsers, you may consider looking into Haskell: as autrijus showed us all with his marvellous Pugs perl6 implementation, Haskell is extremely good at this.

But be warned: haskell is very hard to start playing with :)

Actually, this latest version is written (shhh!) in Ruby. My group is collectively evaluating Ruby and looking for situations in which to test it. Since the SPICE parser is basically an exercise in logical abstraction, it was ideal. Haskell is free with FreeBSD, so I'll take a gander there, too. :) More tools and understanding are always Good Things.

As 5mi11er suggested, my understanding of the problem space has evolved, and P::RD was an important step along that way.

When I was a much younger programmer and had the time to delve into the details, I spent a lot of time reading books about LISP implementation and its underlying theory. Alonzo Church's Lambda Calculus was part of that research. I drove my math profs nuts because none of the rest of the class was prepared to go there, but they wanted to and did. We got a lot of interesting conversation accomplished, but the rest of the class was left very confused and the syllabus didn't get completed. ;)

The nice thing about Perl (or Ruby, for that matter) is that one can do functional programming in it as well as imperative programming. One can look at objects, or one can look at the transition methods and data transformation and information flow. It's all a matter of the level of data abstraction that you model, and the actual language used is immaterial to the understanding embedded in your model.

Haskell does seem to be a very clean way of expressing abstractions. Perhaps, for my parser/equation solver, it would be very suitable. The next stage of our project is to use my solver in conjuction with the analog simulators and statistical correlative software, and that's an iterative situation more suited to the imperative, input-driven nature of a general-purpose language.

Looking at the statelessness of it (Haskell), I can see that that would help us were we to be looking at clustering the solution, but I think that there are enough equations to run with that any real cluster processor I could run this on would be happy solving the equations separately, as there are more equations to solve than there would likely be CPU nodes.

First of all, that the experience of having tried to do it with Parse::RecDescent made the rewrite easier to do.

Secondly I'll bet that if Parse::RecDescent used /\G.../gc matches to tokenize rather than nibbling strings, its performance would have been good enough. Maybe not as good as your final version, but good enough to make do.

Thirdly I'll bet that there are more Perl programmers who are inclined to not use modules that would have helped than there are Perl programmers who are inclined to use modules that are not worthwhile. (Though it is certainly possible to err both ways.)

I was swayed by reports I'd seen from people who swore that P::RD was the cat's meow. You're absolutely right that I gained insight from working with P::RD. Seeing those trace stacks going endlessly through the same fetches and recalculations was mortifying, not edifying! The key to a true RD algorithm is that your early results are saved in the recursive context as you call it again to go deeper. As you come back up the call stack, those results are available.

As for the tokenizer, I actually did a two-pass system, tokenizing the whole string and then working with tokens saved as TTYPE TVALUE. It would have been faster if I had only done one-token lookahead, but the maintenance advantages of separating the tokenizer from the solver outweighed the need for speed. I suspect that the tokenizer is far less a part of the problem than the recalcs. Even with shortcuts, not having access to the actual tokenset stack to replace sub-strings meant that everything was re-fetched again and again.

On your third point, it's a tribute to the languages we have now, especially Perl, that one CAN do so much with the base language. I'm soooo glad to have the choice, and to have the Open Source OSen available to run them, AND to have these multi-gigahertz PCs that labor so under Doze that just fly when set free with open source!!!

  foo = if bar >= 0 then if statistic_p(1, bar/3, 0) > 4 then 1.5u els
+eif statistic_p(1, bar/3, 0) > 2 then 1.2u else 1.0u endif else 0.5u 
+endif
[download]

After that, modules may well exist to fill in some of the coding requirements, but it is often wrong to use a module which misses the mark even slightly. For example, if a module can do everything with a set a files except just one of the things you need, it might be hard to justify doubling the number of passes through all the files just to use the module for most of the requirements, especially if there are hundreds of files adding up to several gigabytes of data to be processed.