Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

Update: Corion ++ pointed out that I reversed the sense of your problem, sorry for my confusion.

Instead, have your wrapper script read 1000 lines from foo, put them in a temporary file and then invoke X. Repeat till done.

#! perl -sw use strict; open FOO, '<', 'foo' or die $!; while( not eof( FOO ) ) { my @lines; push @lines, scalar <FOO> for 1..1000; open TEMPFOO, '>tempfoo' or die $!; print TEMPFOO @lines; close TEMPFOO; qx[ x --filename:tempfoo -firstline:1 -lastline:1000 ]; } close FOO;

Instead of having X supply you 1000 lines at a time, ask for them all and pipe the result to your perl script. In your perl script, read 1000 lines from the pipe into an array, process them, then loop back and get the next 1000 lines. Repeat till done.

The output of X will be blocked while your script processes each batch of 1000 lines. Your script will only ever have to hold 1000 lines in memory at a time. X will never have to backtrack or skip over any lines.

A silly example. (One liner wrapped for display)

perl -ne"print" junk | perl -le" while( not eof(STDIN) ) { $,=' '; push @a, scalar <> for 1..10; chomp @a; print reverse @a; @a=(); }" #Outputs 10 9 8 7 6 5 4 3 2 1 20 19 18 17 16 15 14 13 12 11 30 29 28 27 26 25 24 23 22 21 40 39 38 37 36 35 34 33 32 31 50 49 48 47 46 45 44 43 42 41 60 59 58 57 56 55 54 53 52 51 70 69 68 67 66 65 64 63 62 61 80 79 78 77 76 75 74 73 72 71 90 89 88 87 86 85 84 83 82 81 100 99 98 97 96 95 94 93 92 91 110 109 108 107 106 105 104 103 102 101 ...

The first instance of perl just reads the file junk (one integer per line), and prints it to stdout. The second instance, loops, reading 10 lines, chomping them, reversing them and printing them before emptying the array and going back for the next 10.


Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"Think for yourself!" - Abigail
Hooray!
Wanted!


In reply to Re: splitting an input stream by BrowserUk
in thread splitting an input stream by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (6)
As of 2023-10-03 12:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?