doar4forum has asked for the wisdom of the Perl Monks concerning the following question:

Hi all, Lets say i have a perl script which calls an executable program to run a process. The executable its running continuously for some time and from time to time it writes in a file some results. How can i read those results in real time without having to wait for the executable to finish or without using another script? Thanks :)
  • Comment on parsing the results of the subroutine in real time

Replies are listed 'Best First'.
Re: parsing the results of the subroutine in real time
by friedo (Prior) on Apr 04, 2008 at 20:09 UTC

    Assuming the other process can write to STDOUT, you can use IPC::Open2 to get a filehandle for the output. If you also need to look at STDERR, there's IPC::Open3.

    If the other process has to write to a particular file, see the perlfaq5 entry "How do I do tail -f in Perl?"

Re: parsing the results of the subroutine in real time
by ww (Archbishop) on Apr 04, 2008 at 20:46 UTC

    Knowing what OS would help.

    So would answers to such questions as

    • "Does 'running continuously for some time' mean that it runs to completion soon enough that you won't have to take your eyes off your monitor before that happens (or, alternately, that you can leave your cube at quitting time and read the results in the morning)?
    • "What (plain text? formatted somehow? a database update?) does your executable "print to a file?"
    • "Is that file a temp that's unlinked before your script starts?"

    Please read On asking for help and How do I post a question effectively?.

    Nonetheless, here's a WAG -- barebones, unelaborated pseudocode for near-"real time" reading -- avoiding many OS specific ways to check what the executable is up to, the possibility (probability?) that the output is formatted as something other than simple text, whether the executable locks the file it's writing in increments, etc.

    #!shebang use strict; use warnings; # do ...; # (whatever sets up the call) call executable # (using system, exec, backticks or ....); # First time check for file written by executable if (exists $thefile) { # the file_in_line_13) { readit(); } else { sleep (some amount of time); goto (in some form or another) 18; } sub readit { open '<', EXEOUT, $thefile ; $latest = <EXEOUT>; close EXEOUT; print "$latest \n"; # (to user); recheck(); } sub recheck { my $timesIdential = 1; sleep (time); open '<', EXEOUT, $thefile ; $newlatest = <EXEOUT>; if ( $newlatest ne $latest ) { clearscreen (OS dependant); readit(); } else { $timesIdentical++; if ($timesIdentical = n) { # for some "n" sufficient to convince you # that the executable is done clearscreen; print "Done\n\n" . $newlatest . "\n"; exit(); } } }

    As always, the devil is in the details.

    Update, given the detail (parallel processing) in the reply below.

    Change line 22 to call a sub wherein (given this structure) you'll throw out any prior processing (wasteful and probably NWYW (® "not what you want") and to do your processing (again).

    But we still have no idea of how intensive that processing is, nor how time-critical. Each of those will probably make the replacement suggested in the previous paragraph, in the infamous words of a one-time White House spokesman, "inoperative."

      Hi, First of all..thank U all for the fast responses:) So...i know that a perl script script is executed line by line...the idea is that i dont want to wait for the entire executable to be finished and then read the outfile...i want as the executable its start running to start reading from the file asweel and process the results in parallel...like writing on the same line : execute && read :D P.S. im working in Linux Thanks!
        I think you have some bad information on how Perl programs run. I also think you have some bad information on how input and output work. You might want to read up a bit on how those things happen, and you might want to share that information with whoever told you those things.

        The first set of Perl code here, for example, autoflushes the output buffer after every print, does a substitution, prints, sleeps for a second, then starts with the next line. The second instance of perl just prints everything it gets from STDIN as soon as it gets it.

        perl -ne '$|++; s/print/say/; print $_; sleep 1;' ffi/test.input | per +l -pe ''

        On Linux you actually have some OS-supported options like Unix sockets, named pipes, and such. You can open a file for writing by one process and for reading by another, and the data goes from the one to the other.

        Perl fully handles reading and writing lines of output in other than batch-style modes, though, as it's a general purpose programming language.

Re: parsing the results of the subroutine in real time
by CountZero (Bishop) on Apr 04, 2008 at 20:12 UTC
    What about just opening the file your executable is writing into, quickly grabbing its contents and closing the file? You can then at leisure parse the contents you grabbed.

    You may run into some problems if your executable maintains all the time an exclusive lock on the file, but if that is not the case, it should work.

    Yes, it may be that you are reading at the exact moment the executable is writing to the file and you get only partial input. Your parsing routines should cater for such eventuality by somehow validating the data being parsed. How to do that is difficult to say as we do not know what kind of data in what kind of format is being written by your executable.

    CountZero

    A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James