I'm worried that if I mention Python one more time this weekend I might get dragged off by a lynch mob, but here goes anyway... When you import a module in Python, say module Foo.py, it writes to disk the byte code of the compilation in a file called Foo.pyc. The next time you run, it goes to do the same thing, but checks the time stamps on the .py file and .pyc file, only recompiling the .pyc file if it is stale, and otherwise just reading it in and ignoring the .py file. There are no Makefiles involved.

What is wrong with this idiom? I guess one tricky issue is where you put the .pyc files. If you don't have permission to the directory in which the .py files are stored, and the .pyc files aren't already there, then I suppose you have to figure out where to put the .pyc files you create, though I don't think this would be too terrible. You could have a python cache directory in your home directory, or something like that.

Why wouldn't such an idiom work for Perl?

To be able to do that in Python, you have to give up source filters and prototypes in Perl. If you want no source filter or no prototypes, you know where to find Python, but I prefer that they stay in Perl.

If you want to promise never to use prototypes or source filters (or any module that might use those), you can certainly use B::Bytecode to create compiled ".pmc" files. In fact, I think the current "use" operator already probes for .pmc as well as .pm files.

-- Randal L. Schwartz, Perl hacker
Be sure to read my standard disclaimer if this is a reply.

---

Update: regarding .pmc files: seen in perldoc -tf require:

            Now that you understand how "require" looks for files in t
+he
            case of a bareword argument, there is a little extra
            functionality going on behind the scenes. Before "require"
+ looks
            for a ".pm" extension, it will first look for a filename w
+ith a
            ".pmc" extension. A file with this extension is assumed to
+ be
            Perl bytecode generated by B::Bytecode. If this file is fo
+und,
            and it's modification time is newer than a coinciding ".pm
+"
            non-compiled file, it will be loaded in place of that
            non-compiled file ending in a ".pm" extension.
[download]

I think the bottom line is that mod_perl and similar daemonizing tools (PPerl) makes all this work totally unnecessary. If those didn't exist, people would likely be a lot more interested in saving the in-between steps.

Ah, now we have hit the nail on the head.

If pyton looks at the previous code and its time stamp etc, and other languages complie their code also, then surely this is an advantage if the program code is big and has lots of intergrated snipptes from other modules.
Yet, with perl, its runtime compilation makes it far quicker than the 'compliers' for small programs.

Can a middle man language be concieved, where, for instance it has the option of compiling the code again if its a large program, or performing a runtime execution.

I seem to recall of some tests that tried this approach for perl, and that the resulting binary was so huge, that it turned out that recompiling perl from source was actually faster, than loading this huge binary from disk.

Can anybody confirm or deny this anecdote?

I can confirm that it's an anecdote. {grin}

But I do also recall the same thing, somewhere back in the early perl5 days.

-- Randal L. Schwartz, Perl hacker
Be sure to read my standard disclaimer if this is a reply.

Timestamps fail too often for Makefiles. If there's another comparison possibility, such as hashing the original source file and storing that in a header of the compiled file, it could be useful.

To force caching on all users is a premature optimization, which is an evil step.

I disagree. First, I disagree that just because perl could cache the compiled bytecode that it must do so for everyone. There could be a commandline arg, or maybe a pragma ("use cachedbytecode;") which would enable it. Of course, if there is no downside to the caching (it's smart enough to deal with non-writable filesystems in a reasonable, unobtrusive manner, i.e., disable bytecode caching automatically), then I see no reason not to enable it by default.

Second, I disagree that it's premature optimisation. It's optimisation, yes. But premature? We already know the overhead of compiling code each time. We can compare that to the caching speed. If the cache is slower than recompiling, then we throw away that code, don't commit it to the main trunk, and document it. If the cache is faster than recompiling each time, especially in small programs (where compilation is a larger percentage of the runtime), then it's a proven optimisation.

Finally, I mostly disagree with the OP that this is really even needed. I laugh at my cow-orkers that work in Java. By the time that their code has finished recompiling, my code is almost done executing. And we have similar numbers of lines of code to work with. They won't even have their code loaded into memory by the time mine is finished running, the JVM load time is so slow.

My cow-orkers working in C/C++ are somewhere in the middle .. except that they're all writing JNI code, which again relies on that JVM load time :-)

And hence, what then happens (in other programming languages) is that we make the users specify exactly when and how to cache the translation from the programming language to the intermediate language.

Image-based environments like the Smalltalks and Poplog would be another approach (which has its own benefits and problems of course :-)