Re: Question regarding Tie::File or a better way to handle huge 2-D arrays

Use strings instead of arrays. The following shows that a 100 element array containing strings of 4 million characters takes less that 400MB which is well within the capabilities of most modern machines.

It also shows a subroutine for doing the substitutions that very closely mirror the syntax you have above. And finally, it shows making 300,000 random substitutions takes less than 1 second on my machine:

use Devel::Size qw[ total_size ];;

$a[$_] = '*'x4e6 for 0 .. 99;;

print total_size \@a;;
400003052

sub change { 
    my( $y, $x, $text ) = @_; 
    substr $a[$y], $x, length $text, $text;
};;

print time(); 
change( int( rand 100 ), int( rand 4e6 ), 'test' ) for 1 .. 3e5; 
print time;;
1219523963
1219523963
[download]

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

Comment on Re: Question regarding Tie::File or a better way to handle huge 2-D arrays Download Code

Replies are listed 'Best First'.
Re^2: Question regarding Tie::File or a better way to handle huge 2-D arrays by betterworld (Curate) on Aug 23, 2008 at 21:21 UTC
Use strings instead of arrays. Tie::File uses tied arrays, and (as far as I know) they are implemented in a way that not all of their elements are held in memory at the same time.	[reply]
Re^3: Question regarding Tie::File or a better way to handle huge 2-D arrays by BrowserUk (Patriarch) on Aug 23, 2008 at 22:13 UTC
I'm not sure what Tie::File has to do with my suggestion? Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply]
Re^4: Question regarding Tie::File or a better way to handle huge 2-D arrays by betterworld (Curate) on Aug 24, 2008 at 00:18 UTC
Sorry, I thought you were implying that Benbo's code uses (ordinary) arrays, and I just wanted to point out that his arrays are not ordinary. Anyway, as we can see from Benbo's last note, your suggestion was fine and somehow I have misunderstood both of you..	[reply]