HiredGunTech has asked for the wisdom of the Perl Monks concerning the following question:

I (believe) I am trying to parse from an unterminated string returned from an application.
Enviroment: Ubuntu linux, Eclipse:EPIC as workspace

The application: I am reading my gas meter (by radio) using a Software Defined Radio.
The code for it can be found here: https://github.com/bemasher/rtlamr
Since the output requires the hardware, I'll include a sample below.

The output code looks like this:
10:33:46.213965 decode.go:45: CenterFreq: 912600155 10:33:46.215377 decode.go:46: SampleRate: 2359296 10:33:46.215528 decode.go:47: DataRate: 32768 10:33:46.215651 decode.go:48: ChipLength: 72 10:33:46.215780 decode.go:49: PreambleSymbols: 16 10:33:46.215903 decode.go:50: PreambleLength: 2304 10:33:46.216021 decode.go:51: PacketSymbols: 128 10:33:46.216137 decode.go:52: PacketLength: 18432 10:33:46.216263 decode.go:59: Protocols: scm+ 10:33:46.216404 decode.go:60: Preambles: 0001011010100011 10:33:46.216529 main.go:126: GainCount: 29 {"Time":"2022-02-26T10:35:55.66825429-05:00","Offset":0,"Length":0,"Ty +pe":"SCM+","Message":{"FrameSync":5795,"ProtocolID":30,"EndpointType" +:188,"EndpointID":101419821,"Consumption":305782,"Tamper":2056,"Packe +tCRC":26848}} {"Time":"2022-02-26T10:36:10.667844535-05:00","Offset":0,"Length":0,"T +ype":"SCM+","Message":{"FrameSync":5795,"ProtocolID":30,"EndpointType +":188,"EndpointID":101419821,"Consumption":305782,"Tamper":2056,"Pack +etCRC":26848}} ...


The first eleven lines are only presented when first run, the remaining JSON strings are the actual meter readings, and the information I want to parse.
The results continue forever, so there is no EOF to signal that it's time to process.
I can slurp it in using this code (If you wonder, rtl_tcp has been started in another window):
<br><br> #!/usr/bin/perl -w # Routines to pull data from the Gas meter. # 2/15 jj v0.0.1 # use strict; use warnings; use JSON; use Data::Dumper; use Log::Log4perl qw(:easy); # Initialize Logger my $log_conf = q( log4perl.rootLogger = DEBUG, LOG1 log4perl.appender.LOG1 = Log::Log4perl::Appender::File log4perl.appender.LOG1.filename = /home/jj/Documents/log/Meterlog. +log log4perl.appender.LOG1.mode = append log4perl.appender.LOG1.layout = Log::Log4perl::Layout::PatternLa +yout log4perl.appender.LOG1.layout.ConversionPattern = %d %p %m %n ); Log::Log4perl::init(\$log_conf); my $logger = Log::Log4perl->get_logger(); # sample logging statement $logger->info("this is an info log message"); my @array; my $i=0; my $Radio_string =""; # Need to launch RTLAMR and read the strings coming out. # Can pre-filter to just MY meter on the commandline with: # cd ~/go/bin # ./rtlamr -msgtype=scm+ --format=json -filterid=101419821, 101419999 +-unique=false # Launch the radio system("/home/jj/go/bin/rtlamr -msgtype=scm+ --format=json -filterid=1 +01419821,101419999 -unique=false"); # slurp the headers do { $Radio_string = $_; Print $i; Print $Radio_string; $i++; } until $i == 10; Print("\n-------End of headers.--------\n\n"); # now wait for and process usage reports #@array = json(system("/home/jj/go/bin/rtlamr -msgtype=scm+ --format=j +son -filterid=101419821,101419999 -unique=false")); $Radio_string = $_; Print $Radio_string; @array = json($Radio_string); Dump(@array);


In this code, the code beyond the "# slurp.." never is seen. It may or may not work, I included it to indicate what I was EXPECTING to do.
From this, I would expect that the first couple header lines would be pulled and printed. Then I could get my data.
That's not what I get. It appears as an *unterminated* string coming in "header0\nheader1\nheader2....datastring0\ndatastring\n....." and is printed to the console.

Things I tried:
exec - Doesn't work (by def'n), Perl program waits until command exits.
system - same thing, except a 'fork' is wrapped around exec. Still needs to exit to return a value.
both start a process that I need to 'kill', as rtlamr is orphaned. (I run 'top | grep rtlamr' to find the orphan.)
qx or backtick - same thing.
The archives give a lot of good examples of how to use each these, but in each case, the called program *ends* so the returned data is processed upon the return from the called program.

Tried open:
print "spawned:\n"; chdir ("/home/jj/go/bin/") ; if ( -e "rtlamr" ) {print "Found rtlamr.\n";} $pid = open($handle, "rtlamr -msgtype=scm+ --format=json -filterid=101 +419821,101419999 -unique=false" ) or die "Couldn't open file rtlamr, +$!";

and I got:
spawned:<br> Found rtlamr. Couldn't open file rtlamr, No such file or directory at /home/jj/eclip +se-workspace/ReadGas/ReadFromRadio.pl line 49.


open ( $handle, ">", "rtlamr -msgtype=scm+ --format=json -filterid=101 +419821,101419999 -unique=false") or die "Can't open > output.txt: $!" + ;

Works! (kinda)
So does:
open my $handle, ">", "rtlamr -msgtype=scm+ --format=json -filterid=101419821,101419999 -unique=false" ;
....in that they both 'return' and don't simply hang. But I'm still looking for a return value.
tried open piping the output ie:
open my $handle, "-|", "rtlamr -msgtype=scm+ --format=json -filterid=101419821,101419999 -unique=false" ;
Got this:
spawned: Found rtlamr. loop#:0 Can't exec "rtlamr": No such file or directory at /home/jj/eclipse-wor +kspace/ReadGas/ReadFromRadio.pl line 50. print() on closed filehandle $handle at /usr/lib/x86_64-linux-gnu/perl +/5.30/IO/Handle.pm line 431. loop#:1 ......

Added the 'or die' and found:
Found rtlamr. Can't exec "rtlamr": No such file or directory at /home/jj/eclipse-wor +kspace/ReadGas/ReadFromRadio.pl line 50. Couldn't open file rtlamr, No such file or directory at /home/jj/eclip +se-workspace/ReadGas/ReadFromRadio.pl line 50.


From this, I suspect that 'open' is not really what I want.
I'm not looking at a file, but STDOUT (am I wrong?) from rtlamr.
And it is never really 'opened' and since it is an executable, I don't think 'open' is the right path.

At this point, I think I have exhausted the scrolls.
(Other seekers, check out "Threads, Multi-Processing, and Inter-Process Communication" (https://www.perlmonks.org/?node_id=591896). Good stuff.)
Paths untraveled: IPC::Run, IPC::Open3. Have I missed it?

Please be kind. I have coded, but been on the sidelines for over 20 yrs, and I am rusty.
I have tried to be verbose for the next seeker of knowledge.
Your wisdom is eagerly welcome.

Replies are listed 'Best First'.
Re: Parsing from another program (stream)
by LanX (Saint) on Feb 27, 2022 at 19:14 UTC
      LanX -
      yep, 'Print' is 'print'. I pulled it off to format and some MS-auto-corruptor pretty-print capitalized start words.
      (file-not-found) I could quiet that with a chdir, sometimes. - I tried pointing to the full path, that didn't work. The file is there, but various implementations of 'open' throw that. I see a fair number of other folks fighting that. But I think 'open' is the wrong tool anyway.
      As for 'slurp', not the best term, I get it. The issue I have is there will NEVER be an EOF. I am reading gas usage as reported by the meter. And I can't close the file, either.
      So, is there no other way to parse the output of another application as it is presented??
        > But I think open is the wrong tool anyway.

        I used this approach in the past with great success! 🤷🏽

        Maybe try providing an absolute path to the executable.

        edit

        > The issue I have is there will NEVER be an EOF.

        I expect that you get an EOF at least when the process exits or dies. And this case should be covered IMHO.

        Cheers Rolf
        (addicted to the Perl Programming Language :)
        Wikisyntax for the Monastery

Re: Parsing from another program (stream)
by etj (Priest) on Feb 27, 2022 at 20:11 UTC
    Looking at your various attempts with open statements, you're trying to do inter-process communication (IPC). Take a look at IPC::Open2 or IPC::Open3, to get back a file-handle you can read your data from, handling the various errors that might happen in a more DWIM manner than a raw "open".
      Thanx. Back to the books.
      I suspected I had not found the right path.
        I think the big takeaway is to be ever-lazier (as we know, one of the three cardinal (ha!) virtues of the programmer) - and start by googling to see if someone else has already made the thing I want to do.
Re: Parsing from another program (stream)
by Marshall (Canon) on Feb 28, 2022 at 23:46 UTC
    First this is a cool application! I went to https://github.com/bemasher/rtlamr to get an idea of what you are doing. I didn't read everything, I just perused enough to get the general idea. I assume that you have started rtl_tcp and that it is running in the background somewhere. I further assume that when you start rtlamr in another command window that you get an output as shown at that URL.

    The syntax, "open $fh, '-|', $programName" should work as long as you have the path and file permissions correct (make sure execute permission is enabled). Make sure that rtlamr runs in a command window like the example at the URL.

    Replace this code....

    my @array; my $i=0; my $Radio_string =""; # Need to launch RTLAMR and read the strings coming out. # Can pre-filter to just MY meter on the commandline with: # cd ~/go/bin # ./rtlamr -msgtype=scm+ --format=json -filterid=101419821, 101419999 +-unique=false # Launch the radio system("/home/jj/go/bin/rtlamr -msgtype=scm+ --format=json -filterid=1 +01419821,101419999 -unique=false"); # slurp the headers do { $Radio_string = $_; Print $i; Print $Radio_string; $i++; } until $i == 10; Print("\n-------End of headers.--------\n\n"); # now wait for and process usage reports #@array = json(system("/home/jj/go/bin/rtlamr -msgtype=scm+ --format=j +son -filterid=101419821,101419999 -unique=false")); $Radio_string = $_; Print $Radio_string; @array = json($Radio_string); Dump(@array);
    With this:
    use IO::Handle; #for flush method my $rtlamr = "/home/jj/go/bin/rtlamr -msgtype=scm+ --format=json -filt +erid=1 +01419821,101419999 -unique=false"; open my $fh, '-|', $rtlamr or die "unable to start rtlamr program $!"; my $disk_file = "somepath"; open my $disk_fh, '>>', $disk_file or die "unable to open disk history + file! $!"; while (<$fh>) # this is a "blocking" read of radio receiver { print $disk_fh $_; $disk_fh-> flush; # update disk file immediately } __END__ That's it. The analysis program of the disk file, "somepath" is a separate program.
    As you have correctly surmised, you will not get an EOF while reading the file handle associated with the radio receiver. Although I believe that killing the rtlamr process would cause an EOF, I don't recommend that you do that.

    The loop I show above is infinite - it will never end because there is no EOF. Most of the clock time will be spend just waiting at the while statement with essentially nothing happening otherwise (no CPU cycles for this program). It is possible to do "non-blocking" reads and come up with a scheme to decide that "all the data has been read that can be read at this time". However, that is messy and I don't recommend that.

    You can just leave the above program and rl_tcp running all the time. To do the analysis, you should have a 3rd program.

    In the analysis program, open the disk file that is being appended by the program above. You can read that filehandle, line by line and you will get an EOF when all of the data has been read. There is nothing wrong with reading a file that is being appended by another program.

    However, be aware that very rarely there is the possibility of a premature EOF - meaning that EOF will happen before a newline is encountered. You should allow for that. You can either throw this last line away or just restart the reading process by re-opening the file and reading the file from the beginning again (you just barely missed the last part of the last line).

    So, by writing to an intermediate disk file, we solve the "there is no EOF" problem. The disk file will have an EOF when all the data that is currently in it has been read.
    Also, rather than counting some fixed number of non-Jason lines to throw away, consider a simple filter, a valid line being one that starts with { and ends with \n. If the last line is a partial one, it won't end in \n.