To have a good IDE for Perl 5 is difficult, since Perl 5 has a very complex syntax, especially because Perl 5 depends of runtime states to be really parsed.

I do not believe the later statement to be true. Can you cite credible sources for this assertion?

Of course it is true. The whole reason BEGIN blocks were created was so that useing a module could force the module's run time to happen before the remainder of the useing code's compile time so that the run-time states could impact how the remaining Perl code is parsed.

In the general case, reliably parsing Perl code that uses a module requires one to run the Perl code (and perhaps even XS code) of the used module. This is also true for BEGIN blocks not implied by use statements.

And the PPI documentation covers quite a bit of this. So if you use PPI for implementing your IDE, then your IDE will simply not be able to handle some Perl code. For many people, the Perl code that they deal with is rarely such that PPI actually fails on it (how fully and correctly PPI parses it is surely more variable, though I haven't seen good data on this). In fact, the author of PPI reports that almost everything on CPAN (excluding some things) can be parsed as well as PPI tries to.

But trying to pretend that the parsing problem doesn't in fact exist is just silly, IMHO.

Of course it is true. The whole reason BEGIN blocks were created was so that useing a module could force the module's run time to happen before the remainder of the useing code's compile time so that the run-time states could impact how the remaining Perl code is parsed.

In the general case, reliably parsing Perl code that uses a module requires one to run the Perl code (and perhaps even XS code) of the used module. This is also true for BEGIN blocks not implied by use statements.

Ok, that's a reasonable explanation. Do you know whether there are any examples of this that aren't Source-Filter related issues ?

I believe that modules which change code semantics aren't really relevant to this issue, only those which change how code is parsed for the purposes of simple static analysis.

And the PPI documentation covers quite a bit of this. So if you use PPI for implementing your IDE, then your IDE will simply not be able to handle some Perl code. For many people, the Perl code that they deal with is rarely such that PPI actually fails on it (how fully and correctly PPI parses it is surely more variable, though I haven't seen good data on this). In fact, the author of PPI reports that almost everything on CPAN (excluding some things) can be parsed as well as PPI tries to.

Well that's good. With the size of CPAN, that's a pretty good indicator that a PPI-based approach would be very effective for use in an IDE.

But trying to pretend that the parsing problem doesn't in fact exist is just silly, IMHO.

You're right, of course.

I wonder if anything can be done (perhaps in the area of machine-readable POD) to make it easier for IDE implementors to deal with such issues.

-David

Do you know whether there are any examples of this that aren't Source-Filter related issues ?

Of course there are.

use constant FOO => 1234;
[download]

Whether or not this line is parsed/run first will change how the next line is interpreted:

print FOO;
[download]

This will either be treated as printing a constant, or printing $_ to a filehandle FOO.

And functions with prototypes:

foo +3
[download]

will be parsed differently if foo is declared first as

sub foo ();
[download]

or with

sub foo;
[download]

It means foo()+3 in the former and foo(3) in the latter case.

(It'll do the same if the sub with body is placed in the source code before the call, without the separate declarations, or in a module you use.)

And no source filter in sight.

Of course there are other issues besides source filters. If you read the PPI documentation (really, go ahead, do it now, just the first part, not the really gory details; I'll wait right here in front of the comma until you come back), it covers this fairly well though briefly, even linking to "our own" On Parsing Perl.

Yes, even the non-source-filter issues can impact correctly parsing to the point of doing static analysis correctly.

With the size of CPAN, that's a pretty good indicator that a PPI-based approach would be very effective for use in an IDE.

Um. Maybe. PPI doesn't fail when trying to parse most of CPAN (that is, it doesn't notice that it has parsed something incorrectly to the point that it throws up its hands and declares that it can't go on). I have yet to see any claims as to how accurately or even completely PPI's parsing of all of CPAN actually is.

I wouldn't be surprised to find that many people only rarely run into problems with PPI's parsing of the code that they usually deal with. In fact, if someone is burdened¹ by the use of syntax highlighting when editting their Perl code, then their coding style will be somewhat skewed such that they avoid even reasonable constructs if they happen to confuse their syntax highlighter. So I wouldn't be surprised if they also avoid some things that would trip up PPI. If their syntax highlighter actually uses PPI (such as if they were to use a PPI-based IDE for editting their Perl code), then surely their style will be skewed toward things that PPI gets right.

¹ Is that the right word? I can't find my thesaurus.

My impression is that PPI, in practice, is good enough often enough that a PPI-based IDE could be successful if used on a new project (due to the feedback loop I noted above). The effectiveness of a PPI-based IDE when applied to an existing code base is certainly less certain, IMHO.

- tye        

To have a good IDE for Perl 5 is difficult, since Perl 5 has a very complex syntax, especially because Perl 5 depends of runtime states to be really parsed.

I do not believe the later statement to be true. Can you cite credible sources for this assertion?

Better yet, I can give an example.

$x = foo / bar / - 1;
[download]

Is bar a function call, or part of a regexp? It depends on foo's prototype. If foo is an imported symbol, then running code (the exporting module's import function) is required to determine foo's prototype and therefore whether bar is part of a literal or not.