Re: Tk Scrolled Text and Large Files

I'm scratching my head wondering how you got 50+M of data into the text widget in the first place, which also leads me to wonder whether part of the problem might be how you are loading data into the widget, as well as how you are getting data out of it for storing to a file. (Any chance you might actually be putting three copies of the data in memory at the same time?) In case that could be part of the problem, you might find some interesting stuff on this thread: Displaying/buffering huge text files. (Updated link to point to top of thread, rather than my own reply to it.)

Short of that, there's no reason to create a duplicate of 50+M of data all at once. Treat it like a big input file and loop over the content as you move it to the output file:

open( OUT, ">some_file.txt" ) or die $!;
my $startline = 1;
my $linespan = 100;

while (1) {
    my $startid = sprintf( "%.1f", $startline );
    my $txt = $text_widget->get( $startid, "+ $linespan lines" );
    last unless length( $txt );
    print OUT $txt;
    $startline += $linespan;
}
close OUT;
[download]

Just like in normal file i/o, Tk::Text->get will return an empty string when the requested range falls entirely beyond the end of the current text content.

Comment on Re: Tk Scrolled Text and Large Files Download Code

Replies are listed 'Best First'.
Re^2: Tk Scrolled Text and Large Files by Anonymous Monk on Oct 17, 2005 at 01:19 UTC
I tried hugepad. It looks like a neat utility. Unfortunately it does not work with my kind of data which comes in one single continuous line. I might have to tweak it so that it breaks my text using a different text separator (i.e. local $/ = chr(29)). Here is an example of the data I use: Read more... (2 kB) Reading the data using the 100 linespan seems like a good approach. I had to tweak this line though to get the script to work properly: `my $txt = $text_widget->get( $startid, "$startid + $linespan lines" ).` Thank you guys for leading me in the right direction. Edit: g0n - readmore tags round data	[reply] [d/l] [select]
Re^2: Tk Scrolled Text and Large Files by Anonymous Monk on Oct 17, 2005 at 01:36 UTC
Hmmm! I tried changing the global record seperator but Tk still breaks the lines in its own way within the text widget. Is this hard-coded?	[reply]
Re^3: Tk Scrolled Text and Large Files by graff (Chancellor) on Oct 17, 2005 at 01:50 UTC
Considering the rather strange and messy data you seem to be dealing with, I can't image what sort of line breaking would be "appropriate". Tk::Text lets you control how to handle lines that are too wide for the display window: either no wrapping at all (user must scroll the text display left and right to scan all the text), or wrapping at word boundaries (leaving a jagged edge on the side that isn't being "justified"), or wrapping by character (which might be best for you, considering that you have very long stretches with no apparent "word boundary" (i.e. no spaces or punctuation). In any case, "the global record separator" (whatever you are referring to by that) has nothing to do with line breaking in the Text widget. BTW, PLEASE PUT <code> and </code> AROUND YOUR DATA SAMPLES AND CODE FROM NOW ON. Using any other sort of "tt" or "pre" style markup in your post will tend to really screw up the node display for all of us trying to read your posts.	[reply]