Re: array overhead

When determining the optimum data structure, it is important to know how you will be accessing the data inside, but you have not given many hints as to that.

What are you building them from? How are you using them after? Do you search them? Use them like objects?

The content of arrays is important too. Maybe you can pack them, or concatenate the values into a single string, and split them back out on demand. If you've got way more CPU than memory available, maybe even compress the data. If you're just going to run out of memory regardless, then you have to start saving it to disk.

Comment on Re: array overhead

Replies are listed 'Best First'.
Re^2: array overhead by Anonymous Monk on Jan 05, 2011 at 21:51 UTC
The data in question is "built" by fetching from an external source - either a database, or a TokyoCabinet file - and is financial data - a string for the date, four floats, and an integer. After reading them in they will be accessed (mostly) sequentially in their raw form - as arrays. Basically, I need the arrays to be arrays, but not arrays with so much overhead. I'd prefer to avoid too much CPU overhead if at all possible (speed is a secondary concern), but pack or unpack might be an option.	[reply]

Replies are listed 'Best First'.

Re^2: array overhead
by Anonymous Monk on Jan 05, 2011 at 21:51 UTC

The data in question is "built" by fetching from an external source - either a database, or a TokyoCabinet file - and is financial data - a string for the date, four floats, and an integer. After reading them in they will be accessed (mostly) sequentially in their raw form - as arrays. Basically, I need the arrays to be arrays, but not arrays with so much overhead. I'd prefer to avoid too much CPU overhead if at all possible (speed is a secondary concern), but pack or unpack might be an option.

[reply]