Hacking a source filter into the #! line

dcmertens has asked for the wisdom of the Perl Monks concerning the following question:

I recently wrote a module that extracts my Perl code from a LyX document. (LyX is a WYSIWYG LaTeX editor, if that helps provide some context.) Originally I wrote it as a source filter that was meant to be invoked on the command line, as

perl -MyX my-document.lyx
[download]

(Yes, the module is called yX.pm.) However, it'd be really cool if I could just type

perl my-document.lyx
[download]

Perl is pretty flexible, and I thought that maybe I could get away with throwing a few garbage lines at it before getting to the source filter usage line. So, I tried adding something to the LaTeX preamble, resulting in the following sort of stuff at the beginning of the .lyx file:

#LyX 2.1 created this file. For more info see http://www.lyx.org/
\lyxformat 474
\begin_document
\begin_header
\textclass article
\begin_preamble
%ignore; use yX;
...
[download]

Unfortunately, perl is not quite liberal enough:

Number found where operator expected at test.lyx line 2, near "lyxform
+at 474"
    (Do you need to predeclare lyxformat?)
Backslash found where operator expected at test.lyx line 3, near "\"
    (Missing semicolon on previous line?)
Backslash found where operator expected at test.lyx line 4, near "begi
+n_document
\"
    (Missing semicolon on previous line?)
Backslash found where operator expected at test.lyx line 5, near "begi
+n_header
\"
    (Missing semicolon on previous line?)
Operator or semicolon missing before %ignore at test.lyx line 7.
Ambiguous use of % resolved as operator % at test.lyx line 7.
syntax error at test.lyx line 2, near "lyxformat 474"
BEGIN not safe after errors--compilation aborted at test.lyx line 7.
[download]

The astute observer will notice the very top line begins with a hash character. Furthermore, LyX does not examine the content of this line, so I can change it to something like this:

#!perl -MyX
[download]

Unfortunately, Perl does not let you specify a module in the #! line. :-( Then I struck upon this discussion of the problem on the Perl Porters mailing list. Ric's solution is to use the -d switch, which would look like this:

#!perl -d:MyX
[download]

Lo! it works (after renaming the module, of course)! So my question: If I manage to create a module/extension for LyX that adds this shebang line, would I be a horrible person for exploiting this sort of hack?

Comment on Hacking a source filter into the #! line Select or Download Code

Replies are listed 'Best First'.
Re: Hacking a source filter into the #! line by tobyink (Canon) on Jul 24, 2014 at 15:34 UTC
Absolutely! See also… toby döt ink	[reply]
Re^2: Hacking a source filter into the #! line by mr_mischief (Monsignor) on Jul 24, 2014 at 15:46 UTC
See also Candidate for a new "Evil Uses For Perl" section. which doesn't log you out if your cookie is for a different root from tobyink's. See also What shortcuts can I use for linking to other information?	[reply]
Re^3: Hacking a source filter into the #! line by tobyink (Canon) on Jul 24, 2014 at 15:52 UTC
Apologies… fixed. toby döt ink	[reply]
Re^4: Hacking a source filter into the #! line by mr_mischief (Monsignor) on Jul 24, 2014 at 16:26 UTC
Re^2: Hacking a source filter into the #! line by dcmertens (Scribe) on Jul 24, 2014 at 15:52 UTC
Heh, I'm actually being serious about this. I have found it astoundingly helpful to embed my code among the thoughts, figures, and equations that clarify the code itself. So far, my usual process has been to create a single document with many different, but related ideas, accessible via different commands. By framing the commands in different packages and using inheritance, I can explore evolving concepts for how to solve whatever problem I am working on. For the moment I've rolled my own command system, but I'm considering switching to MooX::Cmd.	[reply]
Re^3: Hacking a source filter into the #! line by tobyink (Canon) on Jul 24, 2014 at 15:56 UTC
Certainly. I'm going to be doing quite a bit of writing code and thoughts related to the code over the next few months. However, my solution will be to keep the code and thoughts in separate files, and scatter "include" tags in the thoughts, so that they can be "compiled" into a single document. toby döt ink	[reply]
Re^4: Hacking a source filter into the #! line by dcmertens (Scribe) on Jul 24, 2014 at 17:20 UTC
Re^3: Hacking a source filter into the #! line by RonW (Parson) on Jul 24, 2014 at 16:16 UTC
A few years ago, we started using Doxygen at work. I rather like it, though I still haven't found a POD to Doxygen filter that works how I would prefer. (Have started yet another, but that ended up morphing into a "Doxy-ish to POD filter".)	[reply]
Re^3: Hacking a source filter into the #! line by Anonymous Monk on Jul 26, 2014 at 00:03 UTC
AKA Literate programming. -> An Introduction to Literate Programming with perlWEB, -> [literate programming] How to mix POD with code? And how to fold this?, etc.	[reply]
Re^4: Hacking a source filter into the #! line by dcmertens (Scribe) on Jul 26, 2014 at 12:13 UTC
Re: Hacking a source filter into the #! line by ikegami (Patriarch) on Jul 24, 2014 at 16:59 UTC
The filter only gets the source that follows the `use` statement (whether inserted by `-M` or present in the source), so you can simply use `#!/usr/bin/perl use yX; ...` [download] For example, `$ cat yX.pm package yX; use Filter::Simple sub { tr/n-za-mN-ZA-M/a-zA-Z/; }; 1;` [download] `$ cat a.pl #!/usr/bin/perl use yX; cevag("Uryyb, Jbeyq!\a");` [download] `$ a.pl Hello, World!` [download]	[reply] [d/l] [select]
Re^2: Hacking a source filter into the #! line by dcmertens (Scribe) on Jul 24, 2014 at 17:23 UTC
For some reason my reply didn't post. The problem with this approach is that LyX expects the very first non-hash-commented line to be "\lyxformat 474" (or some other number). The only lines that I can modify without breaking LyX's ability to load the document are the #-comment lines, so I need some way to instruct Perl to do something within that line. Thus, #! fun. :-)	[reply]
Re: Hacking a source filter into the #! line by dcmertens (Scribe) on Jul 25, 2014 at 02:28 UTC
I have done some digging around LyX's source code and found some very interesting things. First, the LyX parser ignores commas, which means that the following is a valid (beginning of a) LyX source file: `#LyX 2.1 created this file. For more info see http://www.lyx.org/ \lyxformat, 474, \begin_document, \begin_header, \textclass, article, \begin_preamble, %ignore; use yX;` [download] Notice the commas after every element? Those make it possible for perl to parse this document down to the "use yX", at which point the source filter can take over. Unfortunately, I cannot find a way to override how LyX writes the contents of its files. Everything seems to boil down to calling the Buffer class's save method, which does not provide any runtime-modifiable hooks. Perhaps I should take a different approach: defining a "Run Perl Source" output format. :-)	[reply] [d/l]
Re^2: Hacking a source filter into the #! line by dcmertens (Scribe) on Jul 26, 2014 at 11:19 UTC
To expand on the previous idea, LyX can produce lots of different document outputs including DVI, postscript, and PDF through multiple toolchains. I could add a "Run Perl" output document style, which would apply the source filter and run it. The problem is that the code is not run in a terminal (unless you started LyX in a terminal), so I would not be able to feed any arguments to the command. I tried invoking Prima, to see if I could provide a cross-platform mechanisms for getting user input, but encountered trouble. This still may be the best way to do things, but it'll be hard.	[reply]
Re: Hacking a source filter into the #! line by LanX (Saint) on Jul 26, 2014 at 00:53 UTC
Why this complicated and fragile approach with source filters? Why not just writing a real `filter.pl` which extracts and evals the interwoven code (IIRC that's called "tangle" in literate programming) If it was already discussed (tl;dr) please point me there. Please excuse that I'm not that much interested into the details of LyX but if you are free to choose a shebang you should also be able to call `#!filter.pl` directly as alternative interpreter. Cheers Rolf (addicted to the Perl Programming Language and ☆☆☆☆ :)	[reply] [d/l] [select]
Re^2: Hacking a source filter into the #! line by dcmertens (Scribe) on Jul 26, 2014 at 11:12 UTC
I want to use a source filter because I think it's romantic that my LyX file could itself be a Perl script. But I disagree with the premise of your question. Source filters have all kinds of mis-uses, but the string eval approach that you suggest is actually more fragile than writing a source filter. It's not like source filters could trigger segfaults or anything. The code-extraction logic, which is the most fragile part of this whole endeavor, is the same whether working with a source filter or a string-eval script. The difference is that source filters are chained but source filters are not applied to string evals. If my in-LyX code uses source filters (hint: I use PDL::NiceSlice, probably the only syntax-modifying source filter that works properly), a string-eval approach is a far bigger headache that a source filter. I would have to try to scan the code before evaling it, and if I detected a "use PDL::NiceSlice", I would have to apply the string transform code, but only between the "use PDL::NiceSlice" and the "no PDL::NiceSlice", respecting comments and pod cuts. (I've done something similar with App::Prima::REPL.) No thanks. I'll let Perl figure that out for me, by using a source filter on my end. You don't state why you think source filters are fragile, but the typical problem with source filters is that they try to add syntax extensions to Perl, and thus need to do their own Perl parsing. Since no source filter is smart enough to actually parse Perl, it's very easy for the source filter to fall back on dumb regexes. This introduces all kinds of subtle bugs, which cause so many headaches and lost hours that it's usually better to avoid the syntax-modifying source filter. PDL::NiceSlice has had these kinds of bugs filed and fixed, so I trust its operation and use it frequently. The source filter I propose has no substantial edge cases because the LyX-Code delimiters are unambiguous: the content of LyX-Code sections escape material that might have led to ambiguities.	[reply]
Re^3: Hacking a source filter into the #! line by LanX (Saint) on Jul 26, 2014 at 13:09 UTC
If you don't like `eval` (o.O?) extract the code into a separate temporary file and execute that one. It should also be possible to write `filter.pl` in a way that a source filter is applied, but that's getting too esoteric now. I called it "fragile" cause you need "hacks" (sic) to do it (see thread title) while the straight forward approach seems robust. Cheers Rolf (addicted to the Perl Programming Language and ☆☆☆☆ :)	[reply] [d/l]