in reply to Print to Less Screen

How about 

open LESS, '|less -cSRM' or die $!;

my @results = split(/dbDelimeter/,`SQL_Query`);

while(@results) {
    (var1,var2,var3,etc) = splice(@results, 0, 28);
    $var1 =~ s/\n//g;
    $var2 =~ s/oldDateFormat/newDateFormat/;
    $var3 =~ s/moreReplacements/DoneHere/;
    print LESS sprintf (%-10s %-10s %10s,$var1,$var2,$var3);
}

close LESS;

[download]

Update: I just noticed that still reads everything into memory. You want something more like 

open my $query_fh, '-|', 'SQL_Query' or die $!;
open my $less_fh, '|-', 'less -cSRM' or die $!;

while (<$query_fh>) {
   ...
   print $less_fh sprintf (%-10s %-10s %10s,$var1,$var2,$var3);
}

close $less_fh;

[download]

Replies are listed 'Best First'.
Re^2: Print to Less Screen
by bigbot (Beadle) on Jun 04, 2011 at 07:35 UTC
    Wow that's like a whole new way of looking at it. I'll try it now, thanks for the quick reply!
[reply]
Re^2: Print to Less Screen
by bigbot (Beadle) on Jun 04, 2011 at 08:37 UTC
    Wow this Perl piping stuff is mind boggling. I'm looking at the perldoc for "Using open() for IPC".
[reply]
Re^2: Print to Less Screen
by bigbot (Beadle) on Jun 05, 2011 at 01:47 UTC

    My apologies for being all over the place yesterday. I would like to try to better define my problem and what I'm looking to do (the less screen is only a small part of it and really I should have picked a better subject). Here is the program and how it should process:

    1) Take user input and pass it to an external program which does a database query using an SQL statement.

    2) Results from query can be extremely large, so storing them in memory doesn't seem to be a good idea. Either store in a temp file, or pipe the input into the program (if possible).

    3) Do processing on results. Specifically, sorting in different user-defined ways, and regex find/replaces on about half of the fields to clean them up. Also I need to format the results for a decent visual output (which I do using printf).

    4) Output processed data to a less screen for the user. Again, this data can be very large, so I would need to store it in another temp file?

    If anyone can give me a general idea of the best way they think I can do this, I would greatly appreciate it!

    Currently my thoughts are that I need to store the data in a temp file in order to do a sort (couldn't pipe it in?) I would use the Perl module File::Temp and store two temp files- One for the DB results, and one for the results of the sort, regex replacements, and printf formatting.

[reply]

      A BerkeleyDB tied hash temp file might be a good choice. Your hash key would be your sort criteria, the value would ideally be the regexp replaced fields, unless you really need the original data for something else.

      Once you're done reading/storing the SQL results, it's a simple matter to read the hash back in sorted order, doing the printf formatting and less piping then.

      You might also consider saving aside the BerkeleyDB temp files as an expiring cache if you get a lot of common queries that don't necessarily need absolutely up-to-date data, and/or user can specify when they do need the latest data. You're doing a lot of I/O here, it's not gonna be fast. Some users might be willing to trade data currency for response time.

      Is there any specific reason you're not making the SQL query directly from perl? Assuming its possible, that would likely reduce the complexity a great deal, and increase the reliability. Avoiding data problems introduced by parsing raw text output from another program is always good.

      --Dave

[reply]

        Thanks for your response Dave. The reason I am not doing SQL queries directly from Perl is that I do not have permission to access the DB directly. The person who runs the DB has created an intermediary tool that allows me to pass in SQL queries and it then sends back all the results like this:

        Input:

        sql_query_tool "SELECT * FROM table..."

        Output:

        Field1||Field2||Field3||Field4
Field1||Field2||Field3||Field4
...

        [download]

        Also, I have sent my sort criteria to the DB using the query tool, and it works fine. The problem is that certain fields are blank in some cases, so it causes the sort to get screwed up in the following way:

           category     type1    type2     type3

1. fruit        apple    orange    banana
2. fruit        apple              tangerine
3. deserts      pie      cake      icecream

        [download]

        When I try to sort the fields in order of category -> type1 -> type2 -> type3 I have a problem. Sometimes the type fields can be blank. What I want is for these blank fields to be ignored by sort and the next field to be sorted on. Instead what currently happens is the blank field is sorted on which always makes it first. In the example above record 2 is missing the type2 field. When I do the sort it will put record 2 before record 1. Then when I print out the final data it will look like this: 

        fruit    apple    tangerine
fruit    apple    orange       banana

        [download]

        So when printed out tangerine comes before orange, because of that blank field being sorted on. This is why I came to the conclusion that I could not sort using the DB, and needed to parse the data first myself and then sort.

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom