I wrote a piece of code to help you understand how Tie::File works internally:

test.pl:

use Tie::File;
use Data::Dumper;

my $self = tie @array, "Tie::File", "test.pl";
print Dumper($self);
print $array[2];
print Dumper($self);
[download]

If you run it, the result could be (the sample data is from win 98):

$VAR1 = bless( {
                 'autochomp' => 1,
                 'mode' => 258,
                 'deferred_s' => 0,
                 'autodefer_threshhold' => 3,
                 'sawlastrec' => undef,
                 'defer' => 0,
                 'dw_size' => 2097152,
                 'offsets' => [
                                0
                              ],
                 'deferred_max' => -1,
                 'autodeferring' => 0,
                 'recsep' => '

',
                 'rdonly' => '',
                 'memory' => 2097152,
                 'filename' => 'test.pl',
                 'fh' => \*Tie::File::FH,
                 'ad_history' => [],
                 'autodefer' => 1,
                 'deferred' => {},
                 'autodefer_filelen_threshhold' => 65536,
                 'recseplen' => 2,
                 'cache' => bless( [
                                     bless( [
                                              [
                                                0,
                                                $VAR1->{'cache'},
                                                0
                                              ]
                                            ], 'Tie::File::Heap' ),
                                     {},
                                     2097152,
                                     0
                                   ], 'Tie::File::Cache' )
               }, 'Tie::File' );
$VAR1 = bless( {
                 'autochomp' => 1,
                 'mode' => 258,
                 'deferred_s' => 0,
                 'autodefer_threshhold' => 3,
                 'sawlastrec' => undef,
                 'defer' => 0,
                 'dw_size' => 2097152,
                 'offsets' => [
                                0,
                                '16',
                                '35'
                              ],
                 'deferred_max' => -1,
                 'autodeferring' => 0,
                 'recsep' => '

',
                 'rdonly' => '',
                 'memory' => 2097152,
                 'filename' => 'test.pl',
                 'fh' => \*Tie::File::FH,
                 'ad_history' => [],
                 'autodefer' => 1,
                 'deferred' => {},
                 'autodefer_filelen_threshhold' => 65536,
                 'recseplen' => 2,
                 'cache' => bless( [
                                     bless( [
                                              [
                                                1,
                                                $VAR1->{'cache'},
                                                1
                                              ],
                                              [
                                                0,
                                                2,
                                                '

'
                                              ]
                                            ], 'Tie::File::Heap' ),
                                     {
                                       '2' => 1
                                     },
                                     2097152,
                                     2
                                   ], 'Tie::File::Cache' )
               }, 'Tie::File' );
[download]

The size of the read catch and the deferred write buffer might be big, but their upper limit is under control, and you even can set the size by setting the memory option. This is not our major concern.

Now let's look at that hash element called 'offsets'. That's where Tie::File stores, by default, the offsets of file lines, if you don't modify the value of recsep.

Actually the author of this package used memory quite carefully. His code does not populate the offsets array with the offsets of all lines, instead it only populates the offsets of lines up to the last line you accessed.

In our example, at the beginning, the offsets array only contains one element, after we accessed the third line, it only contains three elements.

But still this offsets array is growing, especially in your case, as you are using it as a queue, which means the last line you accessed is the last line of the file, so the offsets of all lines are stored in memory.

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.