If your doing this from C, then you might consider taking both Perl and the C-runtime out of the equation completely and going directly to the OS for your memory.

The OS has an API with which you can reserve a large amount of (virtual) memory without actually allocating it. You then make a second call to actually commit chunks of the reserved allocation as you need it. In this way, you can reserve a contiguous address space large enough to accomodate your biggest likely requirements and then grow the actual allocation into that reserved contiguous address space as you need to. This removes the need for any reallocation and the copying and (temporary) duplication that that entails and bypasses several intermediate levels of freespace chain chasing, coalesing and shuffling to boot.

As your search code is written in C, there is no great problem with utilising the address space provided by the OS for this purpose, you don't need for the overall space to be managed by Perl. You can simple copy your individual array elements to a Perl scalar for returning to the perl code once you have located it.

Needless to say I haven't done this from perl, but the C part is relatively trivial. I located this brief description and sample code on MSDN that demonstrates using this technique if the idea is of any interest.

Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller
If I understand your problem, I can solve it! Of course, the same can be said for you.

In reply to Re: Re: Re: Re: Re: Memory allocation strategies by BrowserUk
in thread Memory allocation strategies by demerphq

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.