As soon as you open up the gates to code that uses things like Filter::Simple, it becomes utterly impossible for anything without a working interpreter to figure out how to parse Perl. And that is an idea which looks to be used more and more aggressively as time goes by.

I agree.

And this is my primary reason for thinking that a "guessy" perl parser would be enough. You eventually reach the point where you are loading a perl interpreter, firing up B:: to do a bunch of analysis, and then start a perl parsing process. I'm not convinced it can EVER be fully done.

And I'm starting to think it more as this thread goes on...

It also brings up questions like how to parse Win32:: code on a Unix box...? How could you parse any arbitrary piece of code without having everything it lists as a requirement installed, or at least on the machine

What if you are using some form of run-time module loading...

The issue of how to account for the effects of execution of code done at compile time is a deep one, with serious consequences for parsing Perl, compiling it, etc. The decision to allow that functionality imposes implicit limitations on Perl.

However my thinking is that since these techniques are becoming more widely used on CPAN, support for doing them are moving into Perl's core, and the plans for Perl 6 are to improve Perl's support for this kind of functionality, I think that trying to ignore this issue is not wise, and will become increasingly not wise as time goes on.

The moral is that you can only get 2 out of 3 of the following desirable features:

1. Easy external parsing and manipulation of code.
2. Support for macros.
3. Syntactic sugar.
Lisp has 1 and 2. TCL 1 and 3. Perl is in the process of adding 2 to 3.

But if I understand some of the docs correctly, even perl itself doesn't really know what everything is, it guesses based on heuristics

But it doesn't guess! It knows precisely. And for you to properly parse Perl, you must emulate those precisely.

There's very little statistical guessing in /usr/bin/perl. About the only two things is the hash-element vs. scalar-followed-by-char-class thingy in a regex, and the "is this a block or a hashref" in certain places that might have either. Everything else is deterministic. Your code must do it right, or it's not parsing Perl as perl would.

Put another way, I know that since sin takes an argument, that if I use slash following it, it's a regex-start. It's never a divide. I can tell that without running it or debugging it. And if I put a double-less-than, it's a here doc. It's never a left shift. But if I replace sin with time, the exact opposite choices are taken.

You cannot guess. To parse Perl, you must know at all times whether you are in a place expecting a value or a place expecting an operator. And to do that, you have to know the prototype of all the built-ins, and how to get the prototype of all the user-defined functions. Which also means you have to step along with the code, executing all the BEGIN blocks, including those spelled u-s-e.

This is not a simple task. Larry admits it. Damian was going to spend the better part of this year working on Parse::Perl as a YAS-funded project. If you are taking it on, but not aware of the things I've posted in this thread, it's a bit like saying "I can fly that plane", but just getting in, without realizing there are clouds and bad weather and other planes, and that landing can be a real pain sometimes, and what happens when the engines go out.

-- Randal L. Schwartz, Perl hacker

Fair enough, you know Perl's behaviour a lot better than I do.

I can see ways of modifying the code towards, and possibly all the way to the behaviour that you would like to see, although I wouldn't like to do so at this point, as I have the level of functionality out of the current codebase that I wanted as a minimum ( excluding bugs ), not to mention that it would slow things down a lot, since I'm not going to write any C into the back end.

But let me add this. Even if the source isn't parsed "properly" in some cases, it may not need to be depending on what you want to do with it.

I have no illusions of taking perl source code and executing it ( "flying a plane" ). But as I have pointed out, there are other uses for perl source that may not need the same level of rigour.

Allow me to add some questions to this.

Does the perl interpreter record all whitespace and comments, the content after the __END__ signal, and other stuff in it's model of the code? All this extra padding is required so that once you have a complete model of the source, you can serialize it all the way back down to text, and come out with what you started with? I would like to be able to have a piece of code walk into the code tree, modify the code within, and then be able to write that code back out to a file.

Is it EVER possible to write a proper and full perl parser without using perl itself... The B modules demonstrate this. Even if you could get a proper parser, what about things like CGI.pm? Can you tell what methods the CGI module will have? At some point, you either have to draw the line, or you end up with perl itself. I choose to draw the line at reading MOST source code. My stardards will probably go up over time, as code is added around the edges that NEED the extra depth.

Finally, two last questions.

1. What were you planning on doing with a perl parser.
2. How slow are you prepared for it to be?

Maybe a name such as Parse::Perl::Guessy or Parse::Perl::Naive or Parse::Perl::Approx or Parse::Perl::Ish or something would be appropriate for your module Adam?

Randal is obviously right that you aren't really parsing Perl without getting everything right in a deterministic fashion. I don't think it'd be good to use up the Parse::Perl name for something that isn't doing this (especially if Damian is on the way to writing something that does).

But I'd certainly like to see a module that makes a decent job of parsing much Perl, for things like syntax highlighting, even if the scope is limited and it can be tricked.

For example, I always put spaces around arithmetic operators. I'd be happy enough for a parser to assume that a “ / ” is always divide, and know that if I ever want a space at the start of a regexp I should use “m/ ” or “/\s” or something.

Smylers