Yes, I cannot imagine that any module of the nature you are dealing with is not expecting the kind of data you have. However the problem was an interesting one to me the longer I studied it. Once I understood the problem, I could not really make any suggestions as such, as I was too busy trying to figure out the solution too. Hence providing a whole solution rather than some pointers.

Did your solution of submitting the full array indices actually work though?

One thought which occured to me and may or not be relevant is that maybe the windows do not need to be 'full', but the array indices just may need to not be empty (or undefined). Without understanding the full layout of the incoming data and the way the data is expecting to be recieved once processed, it is impractical to start guessing what may or not be required though.

The only assumption I did make is that the data you provided is extended a few (hundred?) thousand times. There was a bit of wondering if this was the case or not, or whether you had simplified the data to highlight the problem.

Must the data really be output in the way you have detailed? Or at another angle, is the recipient of the data really expecting the data in the way you are outputing it? You did mention that you are attempting to get a certain module to accept the data. What is it about the data that is not being accepted by the module? What is the expected input/output of that module?

I think you may be able to tell I have become rather inquisitive about this. Such being the nature of scientific research I suppose.

further thought: taking a reference to the @onesarray would likely reduce the overall memory required considerably

my $refonesarr = map '1 1', 0..99;

for (0..170000){
push( @{ $data{$k}}->[$_]}, $refonesarray )
}

print $k," 161:  ",join (',',@${ $data{$k}->[161]}),$/;
[download]

untested code.

Also, I have just now glanced through the previous thread from where you have arrived at this point.

Instead of using the indexes of the hashed array to hold the relevant window why not just enter the 'index' as the first value in the array? a classless class argument held as an array, if you will.

This would reduce the HoAoA to an HoA and would be a lot easier to handle. When you need to extract particular windows on individuals you could then just run a grep search on the classless class value in each individuals array to compile the relevant window arrays for use. This would also eradicate any empty array difficulties. But would this be ok with the differing chromosome coords you mentioned momentarily?

The crass argument may even be better represented as a HoHoA where the keys of the second Hash are string literals representing each numeric window to the array values.

Coyote