Re: searching for unique numbers into a string

just a guess
what happens to $myLine after building it?, 3000000 X 4KB is huge, if you are using that as a plain hash key, your memory may be eaten completely, check another way of solving it.
by building that unique number what you want to achieve?

Vivek
-- In accordance with the prarabdha of each, the One whose function it is to ordain makes each to act. What will not happen will never happen, whatever effort one may put forth. And what will happen will not fail to happen, however much one may seek to prevent it. This is certain. The part of wisdom therefore is to stay quiet.

Comment on Re: searching for unique numbers into a string

Replies are listed 'Best First'.
Re^2: searching for unique numbers into a string by Marshall (Canon) on Apr 07, 2009 at 17:31 UTC
I think these folks are on the right track. The hash is only built for the line being analyzed. Evidently the number of items on a line is small compared with the 3 million line limit. As one hint for HUGE hashes (this situation doesn't qualify from what I understand), but the default hash starts with 8 key "slots". As the hash gets larger, 8,16,32,64,128,etc, all hash keys in the hash have to be re-calculated. If you start getting into hash sizes of like 100,000 keys, this doubling process can cost. It is possible to start a hash with a larger number of buckets than the default of 8, by assigning a scalar value to keys, like keys %hash=2**16 or whatever. If the hash is less than 1024 total key space, this usually doesn't make much difference. Good values for number of available keys is 1/2 expected total number of items in hash (a place to experiment from).	[reply]

Replies are listed 'Best First'.

Re^2: searching for unique numbers into a string
by Marshall (Canon) on Apr 07, 2009 at 17:31 UTC

As one hint for HUGE hashes (this situation doesn't qualify from what I understand), but the default hash starts with 8 key "slots". As the hash gets larger, 8,16,32,64,128,etc, all hash keys in the hash have to be re-calculated. If you start getting into hash sizes of like 100,000 keys, this doubling process can cost. It is possible to start a hash with a larger number of buckets than the default of 8, by assigning a scalar value to keys, like keys %hash=2**16 or whatever. If the hash is less than 1024 total key space, this usually doesn't make much difference. Good values for number of available keys is 1/2 expected total number of items in hash (a place to experiment from).

[reply]