Many moons ago Mike Cowlishaw added a feature to his REXX interpreter that the first time a script was run, it attached the compiled bytecode to the end of the source code. Under (the dos-like) OS/2, this was done but placing the bytecode after the ^Z char that delimited the end of the source module. The second and subsequent times the script was run, the interpreter looked for a signature string after the first ^Z in the file and if found, the bytecode was loaded and run without the need to re-interpret the source. The nice thing about the mechanism besides the quicker startup, was that as (dos) text editors stopped reading once they encountered the ^Z in the source, the byte code was automatically thrown away if the script was edited and saved, and the interpreter would recompile it next time the script was run. A nice simple mechanism of keeping the source and compiled versions in synch--and about the only good use I've encountered for the difference between text/binmode files.

It also made it possible to distribute the 'compiled' version without the source, by using a utility that removed everything between (the rexx equivalent of) the shebang line, and the ^Z. This meant that if the file was edited, the bytecode was discarded automatically, and there was nothing left except the shebang line.

Not that it was that hard to reverse engineer the bytecode, but you invariable ended up with something quite different to the original, and without comments or meaningful variable names meaning that it gave about the same degree of protection as distributing Java bytecode.

I wonder if a similar mechanism couldn't be used for perl scripts. When the script is read, if a paricular byte or series of bytes is encountered followed my a magic-style signature sequence, then Perl would just load the bytecode and ignore the source.

In turn, this would allow the construction and distribution of a light-weight Perl binary that didn't carry the weight of the source code parser and associated stuff, leaving just the bytecode interpreter.

This might restrict some language features, eval and the like, but (to me) that seems a small penalty. Then again, I rarely use eval anyway.

I don't think there is any reason to do it like this for perl6, or if its even possible (for instance, what if the script uses a virtual __DATA__ file?). The run-time engine (code-named parrot) is separate from the compiler/bootstrapper, and already has the ability to emit bytecode (in fact, it is central to the design). Compiling your application into bytecode or an executable will be as simple as hitting a switch on the command line.

After a long thunk, I think I thunked of a solution to the 'problem', though I realise that its only a problem if you see any merit in the idea of keeping the bytecode with the source:)

Basically, if the functionality of Inline::Files gets cleaned up--I've had some problems with it, but that probably me trying to use it in ways it shouldn't be used--and rolled into the Perl 6 base, all we would need is another predefined section eg. __BYTECODE__, and it could be done. Of course, as John points out above the fact that the ^Z trick wouldn't work on most platforms means that we would need to used a checksum of the source embedded in the bytecode section. It might also be unfreindly to have binary data stuffed on the end of the source code, some editors would freek. That could be addressed by encoding it with some simple, fast transformation like Byte 64 or similar.

The Parrot engine would simply look for a __BYTECODE__ section and if it found it decode the Byte64; crc or md5 the file (excluding that section), and if it matched the same embedded in the bytecode section, just load and run it.

I guess what I see as the benefit is avoiding the need for make like tools for ensuring that compiled units match their source code. Seperate files are ok but then you get into the whole game of tracking the seperate elements.

---

Okay you lot, get your wings on the left, halos on the right. It's one size fits all, and "No!", you can't have a different color.
Pick up your cloud down the end and "Yes" if you get allocated a grey one they are a bit damp under foot, but someone has to get them.
Get used to the wings fast cos its an 8 hour day...unless the Govenor calls for a cyclone or hurricane, in which case 16 hour shifts are mandatory.
Just be grateful that you arrived just as the tornado season finished. Them buggers are real work.

Since the text editor won't be automatically throwing away the compiled version upon a save, you don't want to ignore the text. Instead, check a checksum against one in the compiled part to see if it's still the same.

This isn't going the address the mentioned problems. Sure, you can distribute the bytecode. But that means your program shouldn't use Tk, DBI, Date::Calc, or any other XS module. Many programs do use dynamic libraries in one way or another.

Abigail

I agree, being able to capture and store the compiled bytecode would only be a part of the solution. Providing for run-time fix-up of calls to dynamic library routines is another. I can see how this could be done efficiently under Win32 (sorry to pollute your eye-space with that dirty word:), but my knowledge of unix-like systems is insufficient to contribute a solution for them. In the absence of that knowledge, I withheld my ideas.

Perhaps if you could provide some possibilities for runtime fixup for the Unix systems, and we could find someone with expertise of Mac systems, we could between us offer a complete solution to that part of the puzzle?

---

Okay you lot, get your wings on the left, halos on the right. It's one size fits all, and "No!", you can't have a different color.
Pick up your cloud down the end and "Yes" if you get allocated a grey one they are a bit damp under foot, but someone has to get them.
Get used to the wings fast cos its an 8 hour day...unless the Govenor calls for a cyclone or hurricane, in which case 16 hour shifts are mandatory.
Just be grateful that you arrived just as the tornado season finished. Them buggers are real work.

Have a look at PAR.

Best, -gjb-

App::Packer deserves to be mentioned, too.
From its description:

App::Packer packs perl scripts and all of their dependencies inside an executable.

The one thing to worry about is whether this will do you any good. There are fairly enourmous licensing issues involved. It's not tough at all to generate a single bytecode file or executable that, becauseof the licenses on the various modules you use, you can't actually distribute. There's not a whole lot that we can do for you there, though, as that's a legal/social problem, not a technical one.

There are fairly enourmous licensing issues involved.

Can you be more specific? How's it any different from modules sold for other languages like C++ or Java?

—John

With perl right now, licensing issues just aren't. You don't have to worry about the licenses on the modules your code uses if you don't actually ship them with your program, so most people don't even think about the ramifications of the licenses of the modules they use.

Once you start doing all-in-one packages, especially if you're shipping what's essentially a single binary, those license terms become far more important. A not insignificant number of perl modules are pure GPL, which can be an issue both if you don't want to release the source for your program and if you use other modules that use GPL-incompatible licenses. This isn't a problem with the current setup, as the restrictions are on distribution and not use, but once you start distributing...

Not a new issue in general, just one that perl's not had to deal with for the most part up until now.

Goodluck.

Turning a program into a standalone executable won't disable any perl features. The whole of the base runtime will have to be linked in anyway to make sure everything works, so the machinery to handle eval will be there. (Though eval'd code may not run as fast as the compiled code, as it'll have to be evaluated by one of the interpreter cores)

1. Your program will never escape its perl roots. eval (and its kin require and do) will be there, as will the mechanisms to make them work.
2. You'll note the original asked for single file distribution, not necessarily compilation to machine code. The bytecode the engine executes is portable across platforms.