Alright, here's the skinny:

I have to generate a couple hundred million unique codes, that are 16 digits in length, upper-alphanumeric, (insert lots of criteria here). That entire process works perfectly fine, with one caveat; out of memory after I create about 34 million.

All of these codes are being stored in memory, in a hash, to maintain uniqueness. Is there any other method I could possibly use to maintain uniqueness, and not have to store these in memory? I've considered using a database backer, and making the database maintain the uniqueness for me. I haven't implemented that yet, though it seems as though it'd be significantly slower, and speed (since I'm generating 600mill of these) is quite key.

UPDATE:
The reason that I'm not going sequentially, or algorithmically, is that these codes are going to be printed on a product, and then collected and redeemed by consumers of that product, for various things. So, sequentially, or algorithmically, a consumer could redeem codes which they had not purchased the product for, by just using the next incremental number, or figuring out the algorithm.

So, what I've done, is I've just over-generated by a few million, am in the process of tossing them into a database, and I'll just delete what I don't need, or just let them sit there until later. Thanks everybody. Though, I'm still open to suggestions. :)

So, I'm open for ANY suggestions that anybody may have.

-Scott

In reply to Maintaining uniqueness across millions of items by sdbarker

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.