Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
  Platform:
    osname=linux, osvers=2.6.15.7, archname=i486-linux-gnu-thread-mult
+i
    uname='linux terranova 2.6.15.7 #1 smp thu jul 12 14:27:56 utc 200
+7 i686 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dccc
+dlflags=-fPIC -Darchname=i486-linux-gnu -Dprefix=/usr -Dprivlib=/usr/
+share/perl/5.8 -Darchlib=/usr/lib/perl/5.8 -Dvendorprefix=/usr -Dvend
+orlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/
+local -Dsitelib=/usr/local/share/perl/5.8.8 -Dsitearch=/usr/local/lib
+/perl/5.8.8 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/ma
+n3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man
+3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -
+Ud_csh -Uusesfio -Uusenm -Duseshrplib -Dlibperl=libperl.so.5.8.8 -Dd_
+dosuid -des'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemulti
+plicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS 
+-DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE
+_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN 
+-fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='4.1.2 (Ubuntu 4.1.2-0ubuntu4)', gccosand
+vers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1
+2
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
+ lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=/lib/libc-2.5.so, so=so, useshrplib=true, libperl=libperl.so.
+5.8.8
    gnulibc_version='2.5'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'


Characteristics of this binary (from libperl): 
  Compile-time options: MULTIPLICITY PERL_IMPLICIT_CONTEXT
                        PERL_MALLOC_WRAP THREADS_HAVE_PIDS USE_ITHREAD
+S
                        USE_LARGE_FILES USE_PERLIO USE_REENTRANT_API
  Built under linux
  Compiled at Dec  4 2007 08:32:25
  @INC:
    /etc/perl
    /usr/local/lib/perl/5.8.8
    /usr/local/share/perl/5.8.8
    /usr/lib/perl5
    /usr/share/perl5
    /usr/lib/perl/5.8
    /usr/share/perl/5.8
    /usr/local/lib/site_perl
[download]

Amongst others, there is one very strong clue that your perl is limited to 4GB: ptrsize=4,. With 4-byte pointers 2**32/4GB is the maximum memory the process will be able to address. But, a good chunk of that address space will be devoted to the OS. Often the top 1GB.

So if top was showing you that 40% (3.2GB) of memory was used, with some of that occupied by other processes, system drivers etc., the realistic expectation is that your Perl programs will be limited to 3GB.

As for how to resolve the problem, the pat answer would be "use less memory", but that's probably not very helpful. Some possibilities are: build a 64-bit perl; split the data into 2 processes and have the code that uses it do so from a 3rd process via sockets or pipes; store the data more efficiently. It is often possible to reduce the memory usage, but to suggest ways you'll have to give us more information.

• Show us what a couple of the lines of that file look like.

  If they vary widely in length, then the shortest, longest and one that roughly average would be a good start.

• Show us how you are using the hash. That is, once you have created it, what do you do with it?

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

And don't keep truncating the output if you want a resolution

I personally believe that while this suggestion of your is certainly sensible, and the most important one as far as the target of solving his problem is concerned, it would be also recommendable to tell the OP to wrap the output up in <readmore> tags for viewing clarity in the various sections listings.

o wrap the output up in

Read more... (853 Bytes)

I don't know anything about consensus, but I for one find it annoying: at a first glance I recognize it as the output of perl -V and I know I'm only interested in it if I'm interested and knowledegeable enough in the OP's problem to want to delve into it and read it carefully. Otherwise it makes for a longish stream of text to scroll down and find the nodes below it to see if they give a concise answer already. (Funnily enough quite recently someone asked me to remove readmore tags from a post -iirc How not to hardcode a package name?, but I'm not sure- which I think requires them.)

--
If you can't understand the incipit, then please check the IPB Campaign.