Do you know whether there are any examples of this that aren't Source-Filter related issues ?

use constant FOO => 1234;
[download]

print FOO;
[download]

This will either be treated as printing a constant, or printing $_ to a filehandle FOO.

And functions with prototypes:

foo +3
[download]

sub foo ();
[download]

sub foo;
[download]

It means foo()+3 in the former and foo(3) in the latter case.

(It'll do the same if the sub with body is placed in the source code before the call, without the separate declarations, or in a module you use.)

And no source filter in sight.

Both those mechanisms are very predicable in their effect on source code. In those cases, a pre-parse of the source code could yield information which could be easily used by the static analyzer.

Source filters are more difficult because they can have arbitrary effects on the parse of the source code. Possibly some new construct would be necessary to (at least) document the effect of a source filter on the AST.

As tye mentioned, BEGIN {} blocks (and equivalent constructs) are much more problematic, as they are truly computed at runtime. Perl is arguably much more context-sensitive than languages like C# or Java, but with a few simple heuristics a very simple static analysis could be performed.

I also don't see why it's all or nothing... the IDE could get it 99% right and could deal with parser/analyzer warnings in a flexible policy-based (or case-by-case) manner (perhaps with configuration from the user).

I'm not saying it would be easy... just possible. Interest would be the limiting factor.

-David

Both those mechanisms are very predicable in their effect on source code. In those cases, a pre-parse of the source code could yield information which could be easily used by the static analyzer.

That would be quite a pre-parse, even for the simplest cases.

In most cases, though, the parse-affecting statement occurs in another file, whose very location may require executing some of the perl code to determine. Or in XS.

No, it's not dead easy. As ysth said, you have to see what words do, prototype-wise, in other modules.

You typically would have to know which words from that module are getting exported. There could be some system in how it's dealt with, when they're only relying on Exporter, but import is a plain class method. You can implement what it does however you like.

Of course there are other issues besides source filters. If you read the PPI documentation (really, go ahead, do it now, just the first part, not the really gory details; I'll wait right here in front of the comma until you come back), it covers this fairly well though briefly, even linking to "our own" On Parsing Perl.

Yes, even the non-source-filter issues can impact correctly parsing to the point of doing static analysis correctly.

With the size of CPAN, that's a pretty good indicator that a PPI-based approach would be very effective for use in an IDE.

Um. Maybe. PPI doesn't fail when trying to parse most of CPAN (that is, it doesn't notice that it has parsed something incorrectly to the point that it throws up its hands and declares that it can't go on). I have yet to see any claims as to how accurately or even completely PPI's parsing of all of CPAN actually is.

I wouldn't be surprised to find that many people only rarely run into problems with PPI's parsing of the code that they usually deal with. In fact, if someone is burdened¹ by the use of syntax highlighting when editting their Perl code, then their coding style will be somewhat skewed such that they avoid even reasonable constructs if they happen to confuse their syntax highlighter. So I wouldn't be surprised if they also avoid some things that would trip up PPI. If their syntax highlighter actually uses PPI (such as if they were to use a PPI-based IDE for editting their Perl code), then surely their style will be skewed toward things that PPI gets right.

¹ Is that the right word? I can't find my thesaurus.

My impression is that PPI, in practice, is good enough often enough that a PPI-based IDE could be successful if used on a new project (due to the feedback loop I noted above). The effectiveness of a PPI-based IDE when applied to an existing code base is certainly less certain, IMHO.

W.r.t static analysis of perl, there would then seem to be a lot of things that would need to be sorted out (or at least heuristics provided for) to make it useful for the purposes of an IDE.

In fact, if someone is burdened¹ by the use of syntax highlighting when editting their Perl code, then their coding style will be somewhat skewed such that they avoid even reasonable constructs if they happen to confuse their syntax highlighter.

Heh :-) I do use syntax-highlighting wherever possible, if only because it allows me (in my perception) to skim my code faster.

I mostly use Vim (I've been trying to use Emacs of late)... it has lots of problems with its very simplistic highlighter. I don't avoid the "problematic" constructs; I've just formed a habit of inserting comments to "correct" those mistakes ;)

My impression is that PPI, in practice, is good enough often enough that a PPI-based IDE could be successful if used on a new project (due to the feedback loop I noted above). The effectiveness of a PPI-based IDE when applied to an existing code base is certainly less certain, IMHO.

Well, it has to start somewhere (assuming that making perl "more available" to beginners/infidels is a worthwhile goal).

I agree that the feedback loop would eventually lead to a parser that works reasonably for arbitrary perl5 code.

-David.

I agree that the feedback loop would eventually lead to a parser that works reasonably for arbitrary perl5 code.

I think you misunderstood. The feedback loop I see tye talking about is that programmers using syntax highlighting will tend to not use constructs the highlighter gets wrong, thus making the highlighting of the growing codebase closer and closer to perfect. That's the opposite of "arbitrary" perl5 code.