Re: upper or lower triangular matrix to full

Please see How do I post a question effectively? and provide a Short, Self-Contained, Correct Example with a small matrix (say, 10x10) that demonstrates what you are trying to do.

If you're running out of memory just reading the file, then how you might optimize the operations you want to perform depends on what those operations are, which you haven't fully described.

In general, PDL might be of interest.

Comment on Re: upper or lower triangular matrix to full

Replies are listed 'Best First'.
Re^2: upper or lower triangular matrix to full by savita (Novice) on Sep 04, 2017 at 14:39 UTC
Thanks for your response, and for pointing me to the tips on posting- quite new around here and much appreciate it. Here are some more details, to add to my original question. The matrix looks like this: `NA 0.0132 0.0116 0.0082 0.0157 0.0246 0.0000 0.0338 0.0000 0.0173 NA NA 0.0084 0.0042 0.0025 0.0126 0.0061 0.0107 0.0214 0.0177 NA NA NA 0.0159 0.0078 0.0216 0.0133 0.0080 0.0029 0.0067 NA NA NA NA 0.0185 0.0362 0.0167 0.0266 0.0000 0.0093 NA NA NA NA NA 0.0268 0.0046 0.0134 0.0000 0.0000 NA NA NA NA NA NA 0.0063 0.0112 0.0138 0.0000 NA NA NA NA NA NA NA 0.0128 0.0239 0.0000 NA NA NA NA NA NA NA NA 0.0102 0.0188 NA NA NA NA NA NA NA NA NA 0.0088` [download] The elements are tab separated. All I want in the end, is for the NAs to be replaced by the numbers, no other operations need to be done. I need this because a program I want to run requires the input formatted in this way. I will take a look at PDL as sugegsted. Thanks again, Savita	[reply] [d/l]
Re^3: upper or lower triangular matrix to full by etj (Priest) on May 09, 2022 at 13:31 UTC
The first thing to check is that the data consumer (which the OP notes is not them) may be able to actually process the upper-triangular matrix as-is; look at LAPACK, which has routines that operate on triangular matrices directly without needing to copy - in other words, this task may not actually be needed. If it is, the approach I would take using PDL is to pre-allocate a correctly-sized disk-based ndarray (so that RAM is not a limitation) using PDL::IO::FastRaw (untested), from `perldl`: `use PDL::IO::FastRaw; $pdl = mapfraw('fname', {Creat => 1, Dims => [460_000, 460_000], Datat +ype => double()});` [download] It looks like currently PDL::IO::Misc (with `rcols`) and PDL::IO::CSV don't have a facility to read into an existing (possibly disk-based) ndarray. It would still be possible, albeit slow, to read the file line-by-line, probably using Text::CSV_XS, using `slice` to insert each row into the ndarray. Once the ndarray exists, to copy one triangle into the other would be super-easy using PDL::LinearAlgebra::Real#tricpy.	[reply] [d/l] [select]