in reply to Maintaining uniqueness across millions of items

If you get away with "probably unique", you can rely on statistics: by randomly selecting with replacement a small number of elements from a very large set, you can determine the likelihood that all are unique (something like (k-1)!/n^k for k pulls from a set of n).

If you can't get away with "probably", you need to use something sequential (1, 2, 3, ...) or some invertible function of something sequential (f(1), f(2), f(3)...), if you are concerned that these "sequential" numbers don't appear to be sequential. (security-thru-obscurity, beware)

Otherwise, you're stuck with remembering what you've already said: you need a database.

An aside -- there are special approaches for storing "archival" data, where one only needs insert and select, not delete or update. Here's one random link to this idea.

  • Comment on Re: Maintaining uniqueness across millions of items