mantra2006 has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks

I am just curious to know what is the maximum size of the array...
in one of my script am pushing the filenames to an array
which matches to some criteria. After that I am moving
those files which are in array basically matching criteria.
My script is working as expected but just curious to know
if there are say 50,000 filenames then will that array takes those names??

Thanks & Regards
Sridhar

Replies are listed 'Best First'.
Re: Size of the Array
by quester (Vicar) on Dec 07, 2006 at 23:11 UTC
    It's basically limited by the amount of memory on your machine.

    On a Windows XP PC with 1 gigabyte of main memory and a Pentium IV, I can sort two million 24-bytes strings in just over half a minute, but trying to sort four million caused major thrashing (the whole machine slows to a crawl) followed by Perl dying with "out of memory."

    Fifty thousand filenames shouldn't be a problem on reasonably recent machines.

    perl -wle '@a=(1..2_000_000); @b=sort map {scalar gmtime $_} @a; print + "@b[0,1,2,1_999_997,1_999_998,1_999_999]"'
    Fri Jan 2 00:00:00 1970 Fri Jan 2 00:00:01 1970 Fri Jan 2 00:00:02 +1970 Wed Jan 21 23:59:57 1970 Wed Jan 21 23:59:58 1970 Wed Jan 21 23: +59:59 1970 34.42user 0.32system 0:35.76elapsed 97%CPU (0avgtext+0avgdata 29949952 +maxresident)k 0inputs+0outputs (122789major+0minor)pagefaults 0swaps
Re: Size of the Array
by Joost (Canon) on Dec 07, 2006 at 23:21 UTC
    if there are say 50,000 filenames then will that array takes those names??
    Easily. 50,000 entries take about 50,000 * 4 = 200Kb for the pointers + the memory for the filenames, say 200 * 50,000 = 1 Mb plus some overhead. About 2 Mb.

    Perl arrays are implemented as arrays of C pointers so you'll probably run out of memory for the data in it before you reach the maximum length. Theoretically a perl array has a 2**31 = 2147483648 element limit. On a 32 bit perl / os you'll run out of memory for the array itself somewhere before half that amount.

    update: look up "AV" in http://gisle.aas.no/perl/illguts/ if you're really interested.

Re: Size of the Array
by swampyankee (Parson) on Dec 07, 2006 at 23:04 UTC

    The correct answer to "what is the maximum size of the array?" is "lots:" it will depend on the size of the elements (an array of bytes can have more elements than an array of 1 MByte MPEGs) and how much memory is available at that time.

    To answer your specific question: it will probably easily handle those 50,000 names, but this is also tending towards the neighborhood where using a database will be a Good Idea.

    emc

    At that time [1909] the chief engineer was almost always the chief test pilot as well. That had the fortunate result of eliminating poor engineering early in aviation.

    —Igor Sikorsky, reported in AOPA Pilot magazine February 2003.
Re: Size of the Array
by msk_0984 (Friar) on Dec 08, 2006 at 06:27 UTC
    Hi Sridhar

    As our earlier monks said it depends on the Size of the elements and also simnce you are syaing that they are jus file names so hope the size of the elements will not be a big deal for you.

    And also i prefer to use the very much Handy Database to store all the file names becos its always a tedious task to traverse through the array for a partocular filename and its going to take quite a long time for execution and parsing.

    So wat is feel is better go for the Database.......

    depending on how u want it to be ...

    All the best.

    Sushil


    Work Hard Party Harderrr!!
    Sushil Kumar
Re: Size of the Array
by mantra2006 (Hermit) on Dec 08, 2006 at 17:25 UTC
    Thank you Monks for the replies...my doubt is clear now
    ..
    Thanks & Regards
    Sridhar