Limitation of extra large hash

morpheous1129 has asked for the wisdom of the Perl Monks concerning the following question:

Hello Everyone...

I need some serious help here. I'm trying to use a module (Checkpoint.pm from http://www.wormnet.nl/). It goes through the objects.C and configs.C from Checkpoint Configs and loads the data into a hash so I can use it later.

The problem (I think) is that the file is so big that the hash hit's its limit. I see the first 9092 objects in the hash, but everything afterward doesn't get added.

Is this a size limitation of hashes in Perl? I tried this on Unix and Windows and see the same results. Is there a way to get around it? Can I tell Perl to allow extra huge hashes, even if it takes up all my memory?

Here is an example of the structure of a truncated objects.C file (object name and IP have been changed):


(
        :netobj (netobj
                : (objectnameishere
                        :DAG (false)
                        :add_adtr_rule (false)
                        :cp_products_installed (false)
                        :data_source (not-installed)
                        :enforce_gtp_rate_limit (false)
                        :firewall (not-installed)
                        :floodgate (not-installed)
                        :gtp_rate_limit (2048)
                        :ipaddr (10.1.1.1)
                        :type (host)
                        :IPSec_main_if_nat (false)
                )
        )
)

In my file, there are 10572 objects.

Here's the script I wrote to test it:

GetObjects("objects.C"); # Create the Object Hash;

my $line = GetKey('netobj');

foreach (keys(%{$line})) {
	print "$_\n";
}

Here is the bucket details:

6995/16384

I got this by changing my test script to:

GetObjects("objects.C"); # Create the Object Hash;

my $line = GetKey('netobj');
print scalar(%{$line}) . "\n";

Thanks everyone in advance!!

Morpheous1129 (Sean McDaniel)

Comment on Limitation of extra large hash

Replies are listed 'Best First'.
Re: Limitation of extra large hash by BrowserUk (Patriarch) on Apr 02, 2009 at 16:02 UTC
10,000 objects is not a particularly large hash. I've created hashes with millions of keys. That said, your hash seems to be a multi-level HoHoH...which will consume prodgious amounts of space. But like argel, I doubt that space is the problem here. Are you sure that all your 10572 objects are unique? Remember that hashes won't retain duplicates. At any level. if the keys are identical then later values will overwrite easrlier ones. Eg. If you have two objects in your file like: ( :netobj (netobj : (object12345 :DAG (false) :add_adtr_rule (false) :cp_products_installed (false) :data_source (not-installed) :enforce_gtp_rate_limit (false) :firewall (not-installed) :floodgate (not-installed) :gtp_rate_limit (2048) :ipaddr (10.1.1.1) :type (host) :IPSec_main_if_nat (false) ) ) ) ( :netobj (netobj : (object12345 :DAG (true) :add_adtr_rule (false) :cp_products_installed (false) :data_source (not-installed) :enforce_gtp_rate_limit (false) :firewall (not-installed) :floodgate (not-installed) :gtp_rate_limit (4096) :ipaddr (100.10.10.10) :type (client) :IPSec_main_if_nat (false) ) ) ) [download] Even though the details of the object are different, only the last one will be retained in the hash because the names are the same, and the second one will overwrite the first. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply] [d/l]
Re^2: Limitation of extra large hash by morpheous1129 (Initiate) on Apr 02, 2009 at 16:30 UTC
Thanks Guys. There are no duplicates in the objects.C file (just checked). And yes, you're correct - these are hashes of hashes, etc. Also, I forgot to add, it doesn't die with any errors - I just checked the work, and realized some were missing. Perhaps this module doesn't work too well or something. I think I'll have to write my own. Thanks again, -Sean Morpheous1129 (Sean McDaniel)	[reply]
Re: Limitation of extra large hash by almut (Canon) on Apr 02, 2009 at 15:40 UTC
I see the first 9092 objects in the hash, but everything afterward doesn't get added. Is this a size limitation of hashes in Perl? There is no arbitray size limit, only available virtual memory sets a limit (in which case the program would abort with "Out of memory"). The problem almost certainly lies elsewhere.	[reply]
Re: Limitation of extra large hash by cdarke (Prior) on Apr 02, 2009 at 16:00 UTC
The following "Quick and dirty" works for me (Windoze, 5.10): `my @keys = (1..10572); my %hash; @hash{@keys} = undef; print "Size of keys: ".keys %hash,"\n"; print "Bucket details: ".%hash."\n"; keys %hash = 20000; print "Bucket details: ".%hash."\n";` [download] Output is: `Size of keys: 10572 Bucket details: 7718/16384 Bucket details: 8943/32768` [download] Update: Changed to use OPs size of 10572 items.	[reply] [d/l] [select]
Re^2: Limitation of extra large hash by codeacrobat (Chaplain) on Apr 02, 2009 at 19:20 UTC
`keys %hash = 20000; # cool, didn't know that worked :-)` [download] `print+qq(\L@{[ref\&@]}@{['@'x7^'!#2/"!4']});`	[reply] [d/l] [select]
Re: Limitation of extra large hash by Limbic~Region (Chancellor) on Apr 02, 2009 at 19:08 UTC
morpheous1129, You have downloaded a module of the internet that doesn't appear to be on CPAN and the documentation says bug reports should be sent to the author. Have you done that? I have downloaded and looked at the code and it isn't something I would want to try and figure out blind (I don't have an Objects.c file to play with). Cheers - L~R	[reply]
Re: Limitation of extra large hash by ambrus (Abbot) on Apr 02, 2009 at 18:44 UTC
See also Logic trouble parsing a formatted text file into hashes of hashes (of hashes, etc.).	[reply]
Re: Limitation of extra large hash by morpheous1129 (Initiate) on Apr 05, 2010 at 23:34 UTC
Just wanted to send an update: I ditched the module and wrote my own. With the same Objects.C file, I didn't hit any memory limitations. My suspicion is that there is a bug in the module. Thanks for everyone's input and advice!! Morpheous1129 (Sean McDaniel)	[reply]
Re: Limitation of extra large hash by ELISHEVA (Prior) on Apr 02, 2009 at 19:14 UTC
My first question is: how did you check the contents of the hash? Does GetKey() really return one object or some as-large-as-memory-will-allow monster created by an infinite loop? Parsing this file requires a recursive algorithm and recursive algorithms can easily go awry and end up in infinite loops. I'm vaguely suspicious that what is happening is that you have entered into an infinitely loop on the 9092th object and your "data" contains whatever was inserted into the hash before some kind of stack limit was reached - which may be gargantuan and a lot more than 9092 of anything. ~~This is just a guess - given that we can't see any of your code to build the hash. Maybe its worth checking out in any case?~~ Best, beth Update:strike out - didn't notice the citation of the Checkpoint.pm code in OP's original post	[reply]