That's a great point! Currently, there would indeed be an instant where both the ndarray and the input SV needed to have the full amount allocated, in part because get_dataref physicalises its ndarray (i.e. allocates its memory). COW semantics do open a bit of a can of worms, because PDL currently assumes it has full ownership of the block of memory pointed at by the PV. I am going to just hope that all works fine.

PDL's File::Map-using code (via PDL::IO::FastRow) is an alternative approach, which would avoid the use of RAM entirely.

This being a long-standing issue suggests to me there isn't a huge demand for it. However, I am very open to adding a PDL->from_sv method that does what your code does (and would also set the datasv member to the passed-in SV rather than use a char * and SvREFCNT_inc it) - it would also need to deal with the COW situation correctly, which I don't know how to do. Would that help? I think it would actually provide a more generalised implementation of the File::Map stuff in any case.

In reply to Re^7: Reduce RAM required by etj
in thread Reduce RAM required by onlyIDleft

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.