Re^2: Memory efficient way to deal with really large arrays?

in reply to Re: Memory efficient way to deal with really large arrays?
in thread Memory efficient way to deal with really large arrays?

That makes me thing that one can actually calculate the counters without using any memory at all. You only have to sort the array (or arrays) and then traverse it counting consecutive equal elements.

... well, obviously, unless you need to keep the counters in memory for some further processing, but as you have said, the OP hasn't told us yet what his final goal is!

Comment on Re^2: Memory efficient way to deal with really large arrays?

Replies are listed 'Best First'.
Re^3: Memory efficient way to deal with really large arrays? by LanX (Saint) on Dec 13, 2020 at 19:01 UTC
If you keep the pairs, i.e. 2 dim points, how would you want to sort them in one dimension? > the OP hasn't told us yet what his final goal is! Exactly! For instance, if he wanted to process the data sequentially he could also sort them into HoHs or AoHs, since Perl is swapping out unused data. Like this only the currently processed second level structure would be in RAM and the performance overhead for swapping would be negligible. Cheers Rolf _{(addicted to the Perl Programming Language :) Wikisyntax for the Monastery}	[reply]

Replies are listed 'Best First'.

Re^3: Memory efficient way to deal with really large arrays?
by LanX (Saint) on Dec 13, 2020 at 19:01 UTC

> the OP hasn't told us yet what his final goal is!

Exactly!

For instance, if he wanted to process the data sequentially he could also sort them into HoHs or AoHs, since Perl is swapping out unused data.

Like this only the currently processed second level structure would be in RAM and the performance overhead for swapping would be negligible.

Cheers Rolf
_{(addicted to the Perl Programming Language :)

Wikisyntax for the Monastery}

[reply]

In Section Seekers of Perl Wisdom