in reply to Re^3: out of memory problem
in thread out of memory problem

perl -v

This is perl, v5.8.8 built for i486-linux-gnu-thread-multi

Copyright 1987-2006, Larry Wall

[download]

Replies are listed 'Best First'.
Re^5: out of memory problem
by BrowserUk (Patriarch) on Sep 13, 2008 at 11:13 UTC
[reply]
[d/l]
      below is the complete output of "perl -V", thanks. 
      Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
  Platform:
    osname=linux, osvers=2.6.15.7, archname=i486-linux-gnu-thread-mult
+i
    uname='linux terranova 2.6.15.7 #1 smp thu jul 12 14:27:56 utc 200
+7 i686 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dccc
+dlflags=-fPIC -Darchname=i486-linux-gnu -Dprefix=/usr -Dprivlib=/usr/
+share/perl/5.8 -Darchlib=/usr/lib/perl/5.8 -Dvendorprefix=/usr -Dvend
+orlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/
+local -Dsitelib=/usr/local/share/perl/5.8.8 -Dsitearch=/usr/local/lib
+/perl/5.8.8 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/ma
+n3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man
+3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -
+Ud_csh -Uusesfio -Uusenm -Duseshrplib -Dlibperl=libperl.so.5.8.8 -Dd_
+dosuid -des'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemulti
+plicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS 
+-DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE
+_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN 
+-fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='4.1.2 (Ubuntu 4.1.2-0ubuntu4)', gccosand
+vers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1
+2
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
+ lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=/lib/libc-2.5.so, so=so, useshrplib=true, libperl=libperl.so.
+5.8.8
    gnulibc_version='2.5'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'


Characteristics of this binary (from libperl): 
  Compile-time options: MULTIPLICITY PERL_IMPLICIT_CONTEXT
                        PERL_MALLOC_WRAP THREADS_HAVE_PIDS USE_ITHREAD
+S
                        USE_LARGE_FILES USE_PERLIO USE_REENTRANT_API
  Built under linux
  Compiled at Dec  4 2007 08:32:25
  @INC:
    /etc/perl
    /usr/local/lib/perl/5.8.8
    /usr/local/share/perl/5.8.8
    /usr/lib/perl5
    /usr/share/perl5
    /usr/lib/perl/5.8
    /usr/share/perl/5.8
    /usr/local/lib/site_perl

      [download]
[reply]
[d/l]

        Amongst others, there is one very strong clue that your perl is limited to 4GB: ptrsize=4,. With 4-byte pointers 2**32/4GB is the maximum memory the process will be able to address. But, a good chunk of that address space will be devoted to the OS. Often the top 1GB.

        So if top was showing you that 40% (3.2GB) of memory was used, with some of that occupied by other processes, system drivers etc., the realistic expectation is that your Perl programs will be limited to 3GB.

        As for how to resolve the problem, the pat answer would be "use less memory", but that's probably not very helpful. Some possibilities are: build a 64-bit perl; split the data into 2 processes and have the code that uses it do so from a 3rd process via sockets or pipes; store the data more efficiently. It is often possible to reduce the memory usage, but to suggest ways you'll have to give us more information.

        • Show us what a couple of the lines of that file look like.

          If they vary widely in length, then the shortest, longest and one that roughly average would be a good start.

        • Show us how you are using the hash. That is, once you have created it, what do you do with it?
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]
      And don't keep truncating the output if you want a resolution

      I personally believe that while this suggestion of your is certainly sensible, and the most important one as far as the target of solving his problem is concerned, it would be also recommendable to tell the OP to wrap the output up in <readmore> tags for viewing clarity in the various sections listings.

      --
      If you can't understand the incipit, then please check the IPB Campaign.
[reply]
[d/l]
[select]
[reply]

In Section Seekers of Perl Wisdom