As BrowserUk and moritz said, it make no sense to read the whole file into a single variable. Much better to read the file in line by line, discarding the lines you don't need, and storing each line you are interested in in a separate variable. You can keep this list of scalar strings in an array or another data structure.

If each line requires lots of work to process, then your overall problem sounds ideal for Parallel::ForkManager I would try something like:

use strict;
use English;
use Parallel::ForkManager;

# Experiment with this value. I suggest you initially try setting
# it to twice the number of CPU threads you have.
my $MAX_PROCESSES = 8;

$pm = new Parallel::ForkManager($MAX_PROCESSES);

open my $src_FH, '<', 'Huge_source_file.txt' or die "Error opening sou
+rce file $!";
my $line_num = 0;

LINE: while( my $line = <$src_FH> )
{
    next LINE unless 1 == ($INPUT_LINE_NUMBER % 4);
    
    my $worker_pid = $pm->start;
    
    if( $worker_pid )
    {
        # In parent
        next LINE;
    }
    else
    {
        # Call a subroutine to process the line
        process_line($worker_pid);
        $pm->finish; # Terminates the child process
    }
}

[download]

Parallel::ForkManager will maintain a pool of worker threads, and pass processing jobs to each, so you don't need to worry about creating a fork bomb by mistake. If you need to get results back from the worker threads, then there are docs on CPAN explaining how.

In reply to Re: buffering from a large file by chrestomanci
in thread buffering from a large file by cedance

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.