I have no idea how much additional memory Heap::Simple::XS uses under the covers,

For 1e6 items, the memory usage grows from 145MB to over 200MB, which for 10e6 items is going to push a 32-bit machine into swapping.

That said, I think this memory usage may, in part at least, be due to a bug in this incarnation of the code.

I cannot see what would prevent this loop copying everything from %hash into both %known and the heap?

    while (my ($key, $val) = each %hash) {
        next if defined $val;
        if (exists $known{$key}) {
            $hash{$key} = $known{$key};
            next;
        }
        $heap->insert($key);
    }
[download]

Overall, the approach used in the second snippet in Re^2: In-place sort with order assignment seems to be the best. It takes 8 seconds and very little extra memory for 1e6; versus 50 seconds and +25% for the heap. And it happily handles 10e6 in 108 seconds and under 2GB.