A 12 digit hex string can have 281_474_976_710_656 different values, so it's certainly capable of representing 50_000_000 different strings. That said, Digest::SHA1 does not guarantee that two different strings will result in different hashes. Quite the opposite. If all 50M inputs are all different outputs, there's a certain degree of luck involved. (I'm too lazy to compute for you the probability of a collision, but it looks pretty small.)

If you want the output to be exactly as unique as the input, you'll have to come up with some kind of encoding. That is, it will be possible to go from the encoded version back to the original string.

The problem is that your inputs have a significantly larger alphabet than the 16 hex digits, and they're longer than the 12 digits you want to store. (I'm guessing a little here because you haven't told us anything about the format of the inputs.) As such, I think it's safe to say there's no way to encode every possible input string into the output format you want. There just isn't enough "space" to do it in.

That leaves two options, in my opinion.

1. Use a hashing algorithm and rely on luck.
2. Enumerate the inputs with a serial number until you run out of them (i.e., let the database do it).

perl -e 'print ((16**12 >= 50*10**6) ? "Yes" : "No")'

Answer:
Yes

You get max 16**12 = 281474976710656 different values with 12 hex-digits which is far more then 50000000.

Cutting down SHA1 or MD5 to 12-digits will increase the possibility of collisions a lot. Can you not try to use a longer string or use binary?

Update: Sorry, I missed the question about fifty million records. With a 48-bit address space, that gives almost a 99% probability of at least one collision(!) The table in the article shows that 64 bits (sixteen hex digits) gives a one in a million chance of a collision for about sixty one million records.

previously my boss gave me 8-digits, and when I used Digest::MD5, I got duplicated IDs

That's actually not that far fetched.

use Digest::MD5 'md5_hex';

my $x = 'a';
my %found;
my $key;
while (1) {
    $key = substr md5_hex($x), 0, 8;
    if ( exists $found{$key} ) {
        my $first_md5 = md5_hex( $found{$key} );
        my $second_md5 = md5_hex( $x );
        die "found $key at $x ($second_md5) and $found{$key} ($first_m
+d5)\n";
    }
    else {
        $found{$key} = $x;
    }
    $x++;
}
__END__
found a986d9ee at bwma (a986d9ee140c5acbf0d51c00bc5a7810) and kot (a98
+6d9ee785f7b5fdd68bb5b86ee70e0)
[download]

With only eight hex digits, there's only 4_294_967_296 different hashes. You can exhaust that pretty quck.

An md5 hash of a large dataset is always a reduction of information. Different sets of information can result in the same reduction, which is actually the case with md5. There's a paper about md5 collisions.

--shmem

_($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                              /\_¯/(q    /
----------------------------  \__(m.====·.(_("always off the crowd"))."·
");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}

He was using MD5 and then only used the 8 first (or last) characters. This can lead to collisions.

(Also in theory two MD5s from two different inputs can be identical, but this is very unlikely)

You are right. I didn't realise he would take a substring from the result.