in reply to Re: Memory allocation/performance issue for large strings (of binary data) under Windows.
in thread Memory allocation/performance issue for large strings (of binary data) under Windows.

You are correct, my example is rather flawed and in hindsight I could have simplified the example and removed the pack.

The packed structure eventually gets to C code in an XS module. Basically the Perl side is handling the parsing/socket logic and I am using freeze/thaw as a generic message mechanism to serialize perl objects to pass across threads (an event based threading model, sockets and GUI drive event). Some of these objects get 'thawed' in a pure C thread, while others end up in perl threads.

To Answer your direct questions:

Yes, I need all 7 million - it's a single transaction. Batch size is unknown. Sometimes the batch is small (a couple of thousand) to a couple of million. Unknown number of batches, sometimes 1, sometimes hundreds of batches. An end message is sent on socket to end a batch (the server is a 3rd party so I can't change the API).

In hindsight the question should have been "Why is growing a large scalar in windows so slow?"

My results from running the example in linux (perl -V below) 

0.06,0.02
0.4,0.27
2.44,0.27
2.89,0.63
Finished

[download]

and commenting out the line that preallocates: 

0.07,0.01
0.07,0.01
2.43,0.38
2.91,0.76

[download]

not much difference. The windows version is over a 100 times slower (at least with my machine and Perl). Why? 

Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
  Platform:
    osname=linux, osvers=2.2.24-7.0.3, archname=i686-linux-thread-mult
+i
    uname='linux redhat-70-i386.activestate.com 2.2.24-7.0.3 #1 fri ma
+r 14 08:28:25 est 2003 i686 unknown '
    config_args='-ders -Dcc=gcc -Dusethreads -Duseithreads -Ud_sigsetj
+mp -Uinstallusrbinperl -Ulocincpth= -Uloclibpth= -Accflags=-DUSE_SITE
+CUSTOMIZE -Duselargefiles -Accflags=-DNO_HASH_SEED -Accflags=-DPRIVLI
+B_LAST_IN_INC -Dprefix=/opt/ActivePerl-5.8 -Dprivlib=/opt/ActivePerl-
+5.8/lib -Darchlib=/opt/ActivePerl-5.8/lib -Dsiteprefix=/opt/ActivePer
+l-5.8/site -Dsitelib=/opt/ActivePerl-5.8/site/lib -Dsitearch=/opt/Act
+ivePerl-5.8/site/lib -Dsed=/bin/sed -Dconfig_heavy=Config_static.pl -
+Dcf_by=ActiveState -Dcf_email=support@ActiveState.com'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemulti
+plicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS
+ -DUSE_SITECUSTOMIZE -DNO_HASH_SEED -DPRIVLIB_LAST_IN_INC -fno-strict
+-aliasing -pipe -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/inc
+lude/gdbm',
    optimize='-O2',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DUSE_SIT
+ECUSTOMIZE -DNO_HASH_SEED -DPRIVLIB_LAST_IN_INC -fno-strict-aliasing 
+-pipe -I/usr/include/gdbm'
    ccversion='', gccversion='2.96 20000731 (Red Hat Linux 7.1 2.96-85
+)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1
+2
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
+ lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =''
    libpth=/lib /usr/lib /usr/local/lib
    libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lpthread -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=/lib/libc-2.2.4.so, so=so, useshrplib=false, libperl=libperl.
+a
    gnulibc_version='2.2.4'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2'


Characteristics of this binary (from libperl):
  Compile-time options: MULTIPLICITY PERL_IMPLICIT_CONTEXT
                        PERL_MALLOC_WRAP THREADS_HAVE_PIDS USE_ITHREAD
+S
                        USE_LARGE_FILES USE_PERLIO USE_REENTRANT_API
                        USE_SITECUSTOMIZE
  Locally applied patches:
        ActivePerl Build 822 [280952]
        Iin_load_module moved for compatibility with build 806
        PerlEx support in CGI::Carp
        Less verbose ExtUtils::Install and Pod::Find
        Patch for CAN-2005-0448 from Debian with modifications
        Rearrange @INC so that 'site' is searched before 'perl'
        Partly reverted 24733 to preserve binary compatibility
        MAINT31223 plus additional changes
        31324 Fix DynaLoader::dl_findfile() to locate .so files again
        26970 Make Passive mode the default for Net::FTP
        24699 ICMP_UNREACHABLE handling in Net::Ping
  Built under linux
  Compiled at Jul 31 2007 20:53:37
  @INC:
    /opt/ActivePerl-5.8/site/lib
    /opt/ActivePerl-5.8/lib

[download]
  • Comment on Re^2: Memory allocation/performance issue for large strings (of binary data) under Windows.
  • Select or Download Code

Replies are listed 'Best First'.
Re^3: Memory allocation/performance issue for large strings (of binary data) under Windows.
by BrowserUk (Patriarch) on Nov 30, 2009 at 08:38 UTC

    FWIW: I can confirm that I see similar extreme differences between Ubunto/5.10.0 & Vista/5.10.1 :(

    Looking at the differences in compile-time options used: 

    Compile-time options: 

Linux:
THREADS_HAVE_PIDS 
USE_REENTRANT_API 


Windows:
PERL_IMPLICIT_SYS 
PL_OP_SLAB_ALLOC 
USE_FAST_STDIO

    [download]

    Which, if any, of those is the source of the difference ("SLAB_ALLOC"?) will require someone with a much better understanding of the internals to answer.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]

In Section Seekers of Perl Wisdom