richard.sharpe has asked for the wisdom of the Perl Monks concerning the following question:

Hello friends,

having another appetizer: I am reading lines from file, whose name contains backticks, with exact name e.g.:

filename_`date +%d%m%Y`.log

I am providing it to Perl script on (Bash) command line this way, to avoid early backticks execution - I don't want shell to execute them, but Perl, later:

script.pl '"filename_`date +%d%m%Y`.log"'

Note the two levels of quotes - single and double - I don't insist on doing it this way, it was just the first successful try how to force Perl not cutting filename after date, because followed by space.

I am processing that input in the script this way (shortened):

my @in_files = (); foreach my $input_glob (@ARGV) { print $input_glob; push @in_files, glob("$input_glob"); } # reading and storing lines foreach my $in_file (@in_files) { open (LINES, '<', $in_file); ...

After having read, I want to write contents to another file, but now with those backticks executed/interpolated (with current date for purposes of this example).

my $out_file = "/other/path/$in_file"; # this opens file ... open (LOGFILE, '>>', $out_file); # ... but still without backticks execution/interpolation

My question is maybe more general, than illustrated in example above: is there a way (ideally simple) how to instruct Perl to execute backticked substring of some variable and interpolate it with the execution result?

Thank you.

Richard

Replies are listed 'Best First'.
Re: backticks execution "inside" variable (string)
by haukex (Archbishop) on Dec 17, 2019 at 06:33 UTC

    It depends:

    Is it only ever going to be the date/time that you want to interpolate? Then I'd strongly recommend having the user provide a string in strftime format - either the core module Time::Piece, or perhaps DateTime (or even from POSIX).

    Do you care about the potential security issues of your Perl script executing an arbitrary user-supplied command? If not, here's how I might have done it (I'm explicitly calling /bin/bash because of The problem of "the" default shell):

    use IPC::System::Simple qw/capturex/; my $str = q{"filename_`date +%d%m%Y`.log"}; my $out = capturex('/bin/bash','-c',"echo -n $str");

    Another potentially very risky (!) way to do it, if you want to execute arbitrary Perl code instead of the shell's syntax, is eval.

    Otherwise, there might be CPAN modules to do interpolation and execution of commands in the same way the shell does, but I don't know of any off the top of my head. The AM post mentions templates, which might also be a potential generalized solution, depending on what kind of stuff you want to interpolate (typically, templates use some other characters instead of backticks for interpolation, but that's usually configurable).

    Side notes:

    it was just the first successful try how to force Perl not cutting filename after date

    That's the shell doing that, not Perl. Also, in regards to your sample piece of code, see the caveats in To glob or not to glob, and see "open" Best Practices.

      Hello Hauke,

      I am aware that using backticks may be risky (how much, depends on use case context), thank you for your cautions anyway, better measuring multiple times, than doing something ill-judged and then having problems.

      In my case, I am doing automated conversions of log monitoring policies from legacy monitoring system to correlation engine with prescribed correlation templates. Input policies and also correlation templates are using backticks to handle multi-file logs, and it is out of scope of my task, changing these prescribed input and output norms (it would be much more work and this is currently out of project scope), so adapted to these norms, and used backticks also internaly in my conversions and also when generating automated tests. Although currently only dates are in backticks, in general, in general there can be any meainingful command, so I am doing general solution, repeatedly usable also in future, not just covering dates. There is some level of trust, that backticks contents was verified in the past, and also during the analysis before doing automated conversion, and conversions and correlations are running under unprivileged user, so I see the risk level as acceptable in this case, and I am carefully thinking about potential "what ifs" and testing. My showcase was very shortened, removed from its context, so I was interested in very specific thing, not finding alternative solutions what to use instead of backticks, because I know you can't know all the requirements and limitations.

      It is not interesting fo me, finding "who did it" (shell or Perl), I saw, where was the problem (shell was in the service of Perl in this case, so I am in charge to solve this problem by Perl, so I am writing this in Perl forum instead of some shell user group).

      Thank you for all your tips anyway - currently having little time capacity to read thoroughly all articles you provided, I flied them with my eyes and bookmarked.

      May the Perl be with you.

      Richard

Re: backticks execution "inside" variable (string)
by eyepopslikeamosquito (Archbishop) on Dec 17, 2019 at 07:19 UTC

    Not answering your more general question (which I don't understand), but note that you can easily generate a log file name with the current date/time in it with simple Perl. For example:

    sub get_datestamp { my ($s, $m, $h, $d, $mon, $y) = localtime; sprintf('%02d%02d%04d', $d, $mon + 1, $y + 1900); } my $logfile = 'filename_' . get_datestamp() . '.log'; print "logfile='$logfile'\n";
    This code works on any platform and has no need to interpolate bash command lines.

      That +1900 cargo cult is so last century

      use Time::Piece strftime

Re: backticks execution "inside" variable (string)
by kcott (Archbishop) on Dec 17, 2019 at 07:20 UTC

    G'day Richard,

    Passing embedded code in filenames is probably not the best course of action. I suspect what you really want is something more like:

    $ perl -E ' my ($d, $m, $y) = (localtime)[3..5]; say $ARGV[0], "_", $d, $m+1, $y+1900, ".log"; ' filename filename_17122019.log

    However, to answer your specific question, you'd probably want string eval. Do read that documentation closely for all sorts of things to be aware of. It's generally best to avoid string eval unless you have no other recourse. Here's an example of how it might work in your scenario.

    $ perl -E ' my $x = $ARGV[0]; my ($start, $middle, $end) = $x =~ /^([^`]+)(.+?)([^`]+)$/; chomp(my $mid_eval = eval $middle); say $start, $mid_eval, $end; ' 'filename_`date +%d%m%Y`.log' filename_17122019.log

    Note that I didn't need two levels of quotes.

    — Ken

      Thank you, Ken!

      About the first half, my response to this answer also applies, with one exception, that I didn't know [3..5] construct, thank you for that specifically, enriched me by that knowledge.

      But the second is much closer to what I am looking for. Actually I was thinking about splitting and partial evaluation, but I was curious, if there is something yet simpler, what will replace all occurences (theoretically there can be more of them, although currently are not) of backticks pairs with standard output of their contents. I am able to code something like you provided (I don't want to abuse you guys to code that instead of me), this is not the problem, but I prefer re-using already existing functionalities instead of enlarging code base with functionalities, which are already avaliable from some module or built-in construct.

        "... I didn't know [3..5] construct, thank you for that specifically, enriched me by that knowledge."

        You're welcome. I see ++haukex has commented on that construct — I've nothing further to add unless you have questions about it.

        "... but I was curious, if there is something yet simpler, what will replace all occurences ..."

        Personally, I still think your approach to this is questionable; I'll discuss that in more detail further down. One standard way to do multiple replacements is with a substitution using the 'g' modifier (s///g). So, this would achieve what you're questionably asking for:

        $ perl -E ' my $x = $ARGV[0] =~ s/(`[^`]+`)/$1/eegr =~ s/\R//gr; say $x; ' 'DMY_`date +%d%m%Y`_MY_`date +%m%Y`_Y_`date +%Y`.log' DMY_21122019_MY_122019_Y_2019.log

        Note that say and \R became available in v5.10 and the 'r' modifier in v5.14. Depending on your Perl version, you might need something like:

        $ perl -e ' my $x = $ARGV[0]; $x =~ s/(`[^`]+`)/chomp(my $y = eval $1); $y/eg; print "$x\n"; ' 'DMY_`date +%d%m%Y`_MY_`date +%m%Y`_Y_`date +%Y`.log' DMY_21122019_MY_122019_Y_2019.log

        Important: Please be very aware that I am not recommending this approach; I'm merely advising of options.

        The security issues have already been pointed out. In my opinion, it's just far too easy for some bad command to become embedded in your filenames. This could be accidental — a simple typo by your good self — or malicious; however, that's completely immaterial. The fact that it could happen should be sufficient to steer you away from this course of action.

        I would suggest you create a module which does all the work for you and contains tokens representing only those commands that you choose to allow. I would also pick something other than backticks to delimit the command tokens; in the following, I've used tildes as an example alternative but pick whatever you like (it doesn't need to be a single character).

        Here's a rough example of what that module might look like. I've continued using localtime purely for consistency with previous example code I provided; I see lots of alternative suggestions in this thread — pick whatever you want.

        package Pm_11110263_FilenameGen; use strict; use warnings; use Exporter 'import'; our @EXPORT_OK = qw{gen_filename}; my %dispatch = ( 'DATE DMY' => sub { my ($d, $m, $y) = (localtime)[3..5]; return join '', $d, $m+1, $y+1900; }, 'DATE MY' => sub { my ($m, $y) = (localtime)[4,5]; return join '', $m+1, $y+1900; }, 'DATE Y' => sub { return (localtime)[5] + 1900; }, ); sub _perform_substitution { my ($token) = @_; die "FATAL! '$token' is invalid" unless exists $dispatch{$token}; return $dispatch{$token}->(); } sub gen_filename { my ($template) = @_; $template =~ s/~([^~]+)~/_perform_substitution($1)/eg; return $template; } 1;

        Now you only need to use that module and call gen_filename():

        $ perl -E ' use lib "."; use Pm_11110263_FilenameGen "gen_filename"; say gen_filename($ARGV[0]); ' 'daily_~DATE DMY~_monthly_~DATE MY~_yearly_~DATE Y~.log' daily_21122019_monthly_122019_yearly_2019.log

        The 'use lib ".";' is only there because, for demo purposes, I put "Pm_11110263_FilenameGen.pm" in the current working directory. So, after a proper installation of the module, you'd only need two lines of code.

        And if someone (accidentally or maliciously) tried something bad, no harm would be done.

        $ perl -E ' use lib "."; use Pm_11110263_FilenameGen "gen_filename"; say gen_filename($ARGV[0]); ' 'daily_~DATE DMY~_monthly_~DATE MY~_yearly_~DATE Y~_BAD_~RM ROOT~.lo +g' FATAL! 'RM ROOT' is invalid at Pm_11110263_FilenameGen.pm line 26.

        — Ken

        A reply falls below the community's threshold of quality. You may see it by logging in.
        I didn't know [3..5] construct

        kcott's (localtime)[3..5] is a range operator used in a list slice.

        if there is something yet simpler, what will replace all occurences (theoretically there can be more of them, although currently are not) of backticks pairs with standard output of their contents

        I'm still unclear on what syntax you want to use here - the shell's, Perl's, or something custom? In bash's syntax, "X`date`Y" will interpolate, while in Perl's syntax it won't. And the shell quoting rules can get pretty tricky, for example "X`echo \\\``Y""1"'2' will evaluate to X`Y12.

        Perhaps you could show several representative samples of the kinds of strings you want to interpolate, with some complex examples mixed in?

Re: backticks execution "inside" variable (string)
by bliako (Abbot) on Dec 17, 2019 at 10:30 UTC

    eval'ing user-supplied string (the filename) within perl is not only dangerous as others said but it is a practice frowned upon and you will be told off (even if it is innocent) if this is at work.

    I would use a function to parse the user-input/filename and recognise a *few* things that may be useful in a log filename. For example date:

    #!/usr/bin/perl use strict; use warnings; use Time::Piece; my $newfilename = userinput2filename($ARGV[0]); print "got this: $newfilename\n"; sub userinput2filename { my $inp = $_[0]; my $out = $inp; while( $out =~ s/`(.+?)`/_userinput2filename($1)/eg ){} return $out } sub _userinput2filename { my $inp = $_[0]; # date +FORMAT where FORMAT must be understood by POSIX's strftime if( $inp =~ /^date\s+\+(.+?)$/ ){ return Time::Piece->new()->strftim +e($1) } # insert other cases you want to substitute # eventually there is something in backticks that is not in our list die "do not understand '$inp'" }

      Thank you, Bliako, for inspirative command-safeness checking example!

      Although I am afraind, it won't be useful in my case, because in general, there could be anything between backticks, and nevertheless, there is some human-checking in our use case.

      Therefore I am curious, if exists something like this (or I would need to implement it into my codebase myself):

      my $backticketed_string = 'static text `execute this` static text `exe +cute this` ... `execute this` static text'; my $interpolated_string = interpolate_all_backtick_pairs($backticketed +_string);

      It is possible, that there is no such thing in Perl modules or built-ins, and this is not meant to abuse you guys, to do this work instead of me, as said also here.

      Greetings to Island of Saints!

        then try this (with all the bells and whistles for shelling out user input):

        use IPC::Cmd; sub _userinput2filename { my $cmd = $_[0]; print "executing >>unchecked<< user-input : '$cmd'\n"; # see https://perldoc.perl.org/IPC/Cmd.html for more details: my( $success, $error_message, $full_buf, $stdout_buf, $stderr_buf ) += IPC::Cmd::run( command => $cmd, verbose => 1, ); die "failed : $error_message (@$stderr_buf)" unless $success; # this is the stdout of the command return $stdout_buf }

        userinput2filename($string) will run each `command` block in $string and substitute it with the result of command's execution (stdout with all newlines in there) unless there is a failure whereas it dies. Regards to over the hills and far away...

        Edit: it needs the sub (userinput2filename) and main from my previous post.

Re: backticks execution "inside" variable (string)
by Anonymous Monk on Dec 17, 2019 at 01:31 UTC
    Its called a template.