atleastic has asked for the wisdom of the Perl Monks concerning the following question:

could you please tell me if it is possible to store 10^18 array values ?

Replies are listed 'Best First'.
Re: high array values
by hdb (Monsignor) on May 05, 2013 at 19:22 UTC

    Interesting! Let's see:

    • If each array value is only 1 byte, the 10^18 bytes are 10^6 Terabytes.
    • I think we currently have hard disks of maximum capacity of 5TB, so we need 200,000 of those.
    • 5W power consumption when idle make 1MW idle power consumption as a minimum.
    • If we can transfer 6Gbit/s ~ 0.75TB/s, then writing the data will take ~16days.
    So probably not.

    Let's not discuss the 1000 vs 1024 issue.

[reply]
Re: high array values
by dave_the_m (Monsignor) on May 05, 2013 at 19:32 UTC
    The indices on perl arrays are currently 32-bit signed integers, giving a maximum index of 2147483647.

    Dave.

[reply]
Re: high array values
by davido (Cardinal) on May 05, 2013 at 20:25 UTC

    Don't leave just yet. ;) This isn't uniquely impossible for just Perl. But the problem you're trying to solve may be within reach of many languages and architectures. So what problem are you trying to solve?

    Perhaps we've got a communications breakdown. Is it possible you're asking if you're able to store values as high as 10^18 in an array? If that's what you want, yes, and you can do so using Math::BigInt (or other Big* modules).


    Dave

[reply]
Re: high array values
by LanX (Saint) on May 05, 2013 at 19:07 UTC
    In theory it should be possible to create a tied array as a front end which does access data outsourced to files or a DB or a cloud service.

    I'm confident you could find something like that on CPAN.

    The "best solution" depends on the concrete use case...

    edit

    I suppose we are talking about very sparsely populated arrays, cause one byte per element would result in at least 1e6 terabytes of memory consumption...

    Are you sure, you are not rather looking for hashes? =)

    Cheers Rolf

    ( addicted to the Perl Programming Language)

[reply]
Re: high array values
by BrowserUk (Patriarch) on May 05, 2013 at 18:48 UTC

    No.

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
      No.

      Actually, isn't the Utah Data Center supposed to be able to store (and, one presumes, retrieve) data on the scale of yottabytes? Unless I'm getting my exas, zettas and yottas confused, that would seem to do the trick for atleastic's application, whatever it is. (Whether the Utah Data Center is itself a good idea is a completely different discussion.)

[reply]

        I don't know what that is, but certainly google, amazon, cern, the square kilometer array, no doubt NASA and the NSA and many other organisations have storage arrays that are that large; but none of them could construct a single, homogeneous, directly addressable entity of that magnitude.

        (Theoretically, IBM's GPFS could handle it, but only if each element was no more than a single byte. I suspect that the same is true the other distributed filesystems.)

        But even if they could, they could not be accessed from Perl, even via a tied interface, as internally, arrays use 32-bit indexes.

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

Back to Seekers of Perl Wisdom