ghenry has asked for the wisdom of the Perl Monks concerning the following question:

Dear Master Monks,

In Catalyst, LDAP User Auto_complete and arrays with grep I mentioned the amount of users in a Directory at around 500,000 - 1,000,000. This is not uncommon and I have seen up to 1.5 - 2 Million.

So my question is, if during an LDAP search for the auto_complete in the above node, the directory returns say 250,000 users that begin with the letter "g", however unlikely, and the usernames are say a max of 6 letters long, is it possible to determine the rough size of the Array the holds these results?

Kind of 6 x 1 byte = 1 username, x 500,000? Plus what ever Net::LDAP uses to store the results?

2-3MB to be safe?

It's just for a rough idea.

Not sure if I am even on the right track.

Many thanks,
Gavin.

Walking the road to enlightenment... I found a penguin and a camel on the way.....
Fancy a yourname@perl.me.uk? Just ask!!!
  • Comment on Way to predict Array Memory Size for returned LDAP Searches?

Replies are listed 'Best First'.
Re: Way to predict Array Memory Size for returned LDAP Searches?
by g0n (Priest) on Apr 04, 2006 at 10:36 UTC
    To try and cover the overhead from the Net::LDAP::Entry objects I mocked this up: 

    use strict;
use warnings;
use Net::LDAP;
use Net::LDAP::Entry;

my @entrylist;

for (1..500000)
{
    my $entry = Net::LDAP::Entry->new();
    $entry->dn("cn=name$_,ou=test,dc=test,dc=uk");
    $entry->add(cn=>"name$_");
    push @entrylist,$entry;
    print "$_\n";
}
while (<STDIN>){}

    [download]

    which results in the process using 12MB on my (wintel) box. You might want to try it on linux.

    --------------------------------------------------------------

    "If there is such a phenomenon as absolute evil, it consists in treating another human being as a thing."
    John Brunner, "The Shockwave Rider".

    Can you spare 2 minutes to help with my research? If so, please click here

[reply]
[d/l]
Re: Way to predict Array Memory Size for returned LDAP Searches?
by codeacrobat (Chaplain) on Apr 04, 2006 at 10:26 UTC
    Unfortunately stringlength and bytecount are not the same. You can use Devel::Size for estimations on datastructure sizes.
    The example shows "abcdef" take 31 bytes of memory. 
    $ perl -MDevel::Size+'total_size' -e 'print total_size "abcdef"'
31

    [download]
[reply]
[d/l]
Re: Way to predict Array Memory Size for returned LDAP Searches?
by wazoox (Prior) on Apr 04, 2006 at 09:54 UTC
    Well I've tried this : 
    for ( 0..500000) {
push @aa, "abcdef"
}
print "waiting";
$t=<STDIN>

    [download]
    And the perl interpreter gobbles up 22MB. This: 
    print "waiting";
$t=<STDIN>

    [download]
    uses only 1.2MB. So I'd say the hash uses roughly 4MB for 100000 entries (40KB per line).
[reply]
[d/l]
[select]

      :-)

      The easiest way is to mock up a test!

      Common sense, but glad you pointed it out.

      Thanks,
      Gavin.

      Walking the road to enlightenment... I found a penguin and a camel on the way.....
      Fancy a yourname@perl.me.uk? Just ask!!!
[reply]

Back to Seekers of Perl Wisdom