in reply to Perl memory limit - Windows64 strawberry perl

I guess I am dubious about "process some data files, which requires some very large hashes".

A "simple" DB like SQLite probably suits your purpose better than a huge in-memory hash table.
SQLite is actually very sophisticated in how it works.

If you could explain the application in more detail, I am sure that a less memory intensive, and higher performance implementation can be provided.

  • Comment on Re: Perl memory limit - Windows64 strawberry perl

Replies are listed 'Best First'.
Re^2: Perl memory limit - Windows64 strawberry perl
by eyepopslikeamosquito (Archbishop) on Mar 20, 2021 at 06:52 UTC

    A "simple" DB like SQLite probably suits your purpose better than a huge in-memory hash table
    Maybe. We need to learn more about the OP's requirements and algorithms. If the OP's algorithm really does need to perform vast numbers of table lookups, Perl hashes will probably be at least ten times faster than SQLite, as indicated by:

    Note that in High Performance Game of Life (and related Fastest way to lookup a point in a set) profiling indicated that the millions of hash lookups required by the simple GOL algorithm was the dominant performance hot spot ... and the overhead of calling out to (external) SQLite for these was an order of magnitude slower than native Perl hash lookups (so long as you have enough physical memory to comfortably hold the hashes).

    Buk comment: "however fast the C/C++ DB code is, calling into it from Perl adds a layer of unavoidable overhead that Perl's built-in hashes do not have"

[reply]
      I think that we need to learn more about the OP's application.
      I said "probably", not for sure.
      If the hash table won't fit into memory, then we have to do something else.
[reply]

        Nowadays, many home PCs (including mine) have 32 GB of memory and a 64-bit OS ...

        -- from The 10**21 Problem (Part 2) (2014)

        Agreed. Given the OP was "hoping to be able to go to at least 8GB" -- and given I had 32 GB of memory on my home PC way back in 2014 (and I am definitely not a rich man :) -- the simplest and cheapest something else may be simply "buy more memory". Well, that seems attractive for a commercial company because the cost of PC hardware is dwarfed by the cost of people (and the cost/risk associated with making massive code changes to critical production systems).

[reply]

In Section Seekers of Perl Wisdom