deepblue has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,

I am a user of Expect.pm module. I am in love with Expect.pm.

Recently I came across a situation wherein I felt the need for support for a single line (m/<RegEx>/s) regular expression matching when the arguments to expect are similar to:

$object->expect ($timeout, ‘-re’, <some pattern>);

The situation wherein I felt this need follows –

I have a shell script to install software on UNIX. This script keeps on prompting the user for answers to its questions. The user then provides the answers.

Between 2 consecutive questions, there is some multiple line text printed on screen as shown in the snippet below. The next question is displayed once the current one is answered. And the script goes this way when run manually.

So, every time, the last line in all the multi-line text displayed is the line having question. And I have to get this question using regular expression or anything similar so that I can provide the corresponding answer using some pattern matching. If this is done using the normal multi-line matching regular expression feature provided ($Expect::Multiline_Matching), this matches many intermediate lines and thus the automation script is not that efficient and fails at a later stage.

Instead, if a single line matching support is provided, the last line can be directly extracted using a regular expression as simple as:

qr/^(.*)\n(.*)$/s

Also, in all the user interaction programs, the last line would be a question / prompt. Hence this enhancement would greatly simplify the automation script’s design.

Please Note: The text that is a question is the question the user has to answer, and the text after that question is the corresponding user response/answer.

Snippet from the install script:
-------------------------------------------

Bash-2.05$ ./Install.sh This software uses the MD5 message digest algorithm for authentication. The source code used to implement this function is derived from the RSA Data Security, Inc. MD5 Message Digest Algorithm, published in RFC 1321. Are you ready to proceed? (yes/quit) [yes]: Y Software Installation ======================= What is the Unix username? [root] rkabra What is this host's Domain Name? xyz.com Software will be installed with the following settings: ======================================================== Unix username rkabra This host's DNS Domain Name xyz.com This will be a simplified install no Do you accept these settings? (Yes/No/Quit) [yes] Y
……. and so on…

Availability for multi-line matching is already present. Users of Expect.pm would really appreciate if support for single line match is also incorporated / added.

Is there any other way (a workaround) to overcome the problem I am having? If yes, could you please let me know?

Thanks in anticipation.

Regards,
~DeepBlue

Replies are listed 'Best First'.
Re: Need a solution to Expect.pm 's single line regex matching limitation
by RaduH (Scribe) on Nov 16, 2007 at 20:33 UTC
    Hi DeepBlue,

    I'm not sure I understand what is the problem exactly. I mean your explanation is coherent but when I use expect to handle a SFTP session, this line:

    #$sftp->expect(5, 'sftp>') || die("Never got sftp prompt on $targetSer +ver, ".$sftp->exp_error()."\n");
    Is waiting for the SFTP prompt and then I send the upload command and then I do:
    #$sftp->expect(120, 'sftp>') || die("Never got sftp prompt on $targetS +erver after upload, ".$stfp->exp_error()."\n");
    to capture the end for the transmission. Whatever else gets printed is ignored and my second line only intercepts the new SFTP prompt which is pretty much single(last) line match from all the output the upload session generates. The reason is say I don't understand your problem is that I didn't do anything special for this. I simply ignore everything until I find what I expect (don;t have to ignore explicitly, just look for that you know prompts you for input).

    If your regular expression you use matches some of the intermediary blurb spit out before your get prompted for input again, maybe you can change your regexp. If you have access to the install script and can modify it, maybe you can mark your prompt lines with a distinctive patters that'll help you write a regexp that won't give you false intermediary positives.

    I think the bottom line is you don't ignore explicitly just match what you're looking for and try to eliminate false positives.

    Hope this helps (or I totally didn't understand your problem :) )

Re: Need a solution to Expect.pm 's single line regex matching limitation
by Argel (Prior) on Nov 17, 2007 at 01:23 UTC
    I have to agree with RaduH that I do not understand the problem. Based on the sample output I'm not even sure if I understand why you are using a Regex. If you know all the prompts you will get then you could pass an array in instead and then check to see what you matched against. Or if you know the order just specify the one you expect (e.g. 'What is the UNIX username').

    In Re: Expect question I gave some example code that might help you out. I'll repeat the relevant portion here.

    print $expect_instance "$username}\r"; ( $which, $why, $match, $before, $after) = $expect_instance->expect( $ +timeout, 'Password:'); if( ! $which ) { myerror( 'send the password:'.$why );
    $before is what came before the match, $match is what it matched on (useful if you passed in more than one match candidate), and $after is what came after the match. Note that $before and $after are strings and they may contain newlines..
    If you want to do multiple matches then I think just replace 'Password' with e.g. @possible_matches in the example code.

    I get the impression you are trying to do things beyond the scope of Expect.pm. You should focus on improving the accuracy of your matches and then performing string operations and/or regexes on $before, $after, and $match if appropriate.

    If you are still having issues then could you post some sample input/output that shows the problem and what code you are using.

      Dear Monks,

      Thanks for the quick responses. :)

      Now I understand that I am not clear. Please let me explain the problem in detail.

      Also, I am sorry if this explaination goes a bit detail or lengthy.

      Pictiorilly:

      Installation Script <----> Perl Script(Automation Script) <---- Configuration File (storing question and their answers)

      I have an installation script that installs a software on UNIX.

      This script while in execution prompts for many (more than 200) questions and the user has to answer them. All the questions are not asked in a linear/serial way such that, say if a question asks for:
      "which database to use?"

      (supported are Oracle, DB2, MySql), and if the user selects MySql, the questions that follow are different than those that would have been asked for Oracle DB selection.

      In Programming's terms, it is like:

      if (DB == 'ORACLE') { ask Oracle set of questions } else if (DB == 'MySql') { ask another set of questions related to MySql DB. (These questions are different than those asked by Oracle's select +ion) }

      Hence, in short, there are multiple code paths to follow, based upon the selection of database, OS (Linux, Solaris, AIX ...) and so on...

      So, if I write the Perl automation script in a way that all the prompts (more than 200) are literally written to expect the prompts, my automation perl script would go beyond 5000 lines and would be cumbersome/hard to modify and maintain.

      Also all the questions/prompts are not literal.

      For example:
      ----------------------------------------------------------------------------
      If I respond to a question:

      What is the Unix username? [root]
      as: "deepblue" the following question(s) would have this "deepblue" into its text as:
      What is the home directory for deepblue? [/users/deepblue]
      ------------------------------------------------------------

      and if I send the response as "deepfritz"

      the question would have "deepfritz" into it's text as:

      What is the home directory for deepfritz? [/users/deepfritz]
      ----------------------------------------------------------------------------

      So I cannot expect on literal strings here. This is another limitation that I cannot do a simple:

      $object->Expect(<Some Text>);

      in my Perl Automation script. Hence, I must do a Regular Expression matching.

      So I thought of organising this whole perl automation program such that:

      1. All the generic prompts/question are written into a single file like:
      PROMPT 1 = DEFAULT_Answer1
      PROMPT 2 = DEFAULT_Answer2
      PROMPT 3 = DEFAULT_Answer3
      PROMPT 4 = DEFAULT_Answer4
      PROMPT 5 = DEFAULT_Answer5

      By generic prompts I mean: The fixed text in a prompt/question.

      2. The Perl automation script would read this file and load it into the memory.

      3. Then this perl automation script would scan/extract the question using Expect's RegEx facility from the installer script using $object->Expect() and do a reqular expression matching with the PROMPTs loaded into memory and then send the corresponding DEFAULT_Answer to the installer script (invoked using $object->Spawn())

      This would make the Perl script quite small and easy to understand, modify and maintain.

      Also, all the data would be clubbed together into one place (in the Prompt's file) and the code in the Perl Automation script.

      So, a pseudo code for the script would now look like:
      --------------------------------------------------------------

      Spawn(Install Script);

      Load the PROMPT's file into memory;

      while (Install Script does not complete execution)
      {
      Read the questions from the install script;
      Do a regular matching with the PROMPTs stored;
      Send the corresponding ANSWER to the Install Script;
      }

      Instead of many consecutive prompts for each and every question like:

      --------------------------------------------------------------
      Spawn(Install Script);

      Load the PROMPT's file into memory;

      expect (Prompt 1);
      send (Answer to Prompt 1);

      ...
      ...

      if (expect (Prompt 100))
      send (Answer to Prompt 100);

      # And follow a different code path.

      if (expect (Prompt 150))
      send (Answer to Prompt 150);

      # And follow a different code path.

      expect (Prompt 200);
      send (Answer to Prompt 5);

      END;

      --------------------------------------------------------------

      Also, all the prompts are always in the last line of the multiple line text from the install script.

      To summarise:
      1. The prompts/questions are not literally/exactly the same always. Questions contain some variable text that is formed at runtime.
      2. Expecting on literal strings would make the Perl Automation Script un-necessarily very lengthy, hard to understand, modify and maintain.

      Does this make sense to all of the Monks there ?

      If there is some other efficient way to handle this, suggestions are a great WELCOME!!!

      Please help me in solving this.

      Eagerly waiting for your replies... :)

      Regards,
      ~DeepBlue

        It's all in the Expect documentation.

        Sample install script:

        #!/usr/bin/perl my %questions = ( 'Path for package: ' => <<EOH, So we are poing to install The Best All-Purpose Compiler ever. It groks perl, java, c#, haskell, forth and lisp and many more all at the same time. Big deal. EOH 'do you want Visual Basic support? ' => <<EOH, The next abomination is optional. PLEASE think again if you are inclined to install support for that cruft. EOH 'What is your nick? ' => <<EOH, We want some sensitive information from you to harrass you on every IRC channel in which you talk bad of our product. Yes, we monitor them all. EOH ); for (keys %questions) { print $questions{$_}; sleep 1; print; chop ($foo = <STDIN>); sleep 1; print "you said: '$foo', good. We'll come back to that.\n"; } print "Thank you for using The Great Foo.\n";

        expect script to deal with that install script:

        #!/usr/bin/perl use Expect; my %answers = ( 'Path for package: ' => '/foo/bar/quux', 'do you want Visual Basic support? ' => 'NO!', 'What is your nick? ' => 'shmem', 'Thank you' => '', ); my $exp = Expect->new(); $exp->spawn( 'perl', 'install.pl') or die; my $prompts = join('|', map {qr{\Q$_\E} } keys %answers); $exp->expect(120, -re => $prompts, sub { my $exp = shift; my $matched = $exp->match; my $answer = delete $answers{$matched}; $exp->send( $answer."\n"); $exp->exp_continue if keys %answers; } ); print "All done.\n";

        Output:

        So we are poing to install The Best All-Purpose Compiler ever. It groks perl, java, c#, haskell, forth and lisp and many more all at the same time. Big deal. Path for package: /foo/bar/quux you said: '/foo/bar/quux', good. We'll come back to that. We want some sensitive information from you to harrass you on every IRC channel in which you talk bad of our product. Yes, we monitor them all. What is your nick? shmem you said: 'shmem', good. We'll come back to that. The next abomination is optional. PLEASE think again if you are inclined to install support for that cruft. do you want Visual Basic support? NO! you said: 'NO!', good. We'll come back to that. Thank you for using The Great Foo. done.

        That's how it works, basically. Of course, you'll have to tweak your regexes well.

        --shmem

        _($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                                      /\_¯/(q    /
        ----------------------------  \__(m.====·.(_("always off the crowd"))."·
        ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}