You are correct, my example is rather flawed and in hindsight I could have simplified the example and removed the pack.

The packed structure eventually gets to C code in an XS module. Basically the Perl side is handling the parsing/socket logic and I am using freeze/thaw as a generic message mechanism to serialize perl objects to pass across threads (an event based threading model, sockets and GUI drive event). Some of these objects get 'thawed' in a pure C thread, while others end up in perl threads.

To Answer your direct questions:

Yes, I need all 7 million - it's a single transaction. Batch size is unknown. Sometimes the batch is small (a couple of thousand) to a couple of million. Unknown number of batches, sometimes 1, sometimes hundreds of batches. An end message is sent on socket to end a batch (the server is a 3rd party so I can't change the API).

In hindsight the question should have been "Why is growing a large scalar in windows so slow?"

My results from running the example in linux (perl -V below) 

0.06,0.02
0.4,0.27
2.44,0.27
2.89,0.63
Finished

[download]

and commenting out the line that preallocates: 

0.07,0.01
0.07,0.01
2.43,0.38
2.91,0.76

[download]

not much difference. The windows version is over a 100 times slower (at least with my machine and Perl). Why? 

Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
  Platform:
    osname=linux, osvers=2.2.24-7.0.3, archname=i686-linux-thread-mult
+i
    uname='linux redhat-70-i386.activestate.com 2.2.24-7.0.3 #1 fri ma
+r 14 08:28:25 est 2003 i686 unknown '
    config_args='-ders -Dcc=gcc -Dusethreads -Duseithreads -Ud_sigsetj
+mp -Uinstallusrbinperl -Ulocincpth= -Uloclibpth= -Accflags=-DUSE_SITE
+CUSTOMIZE -Duselargefiles -Accflags=-DNO_HASH_SEED -Accflags=-DPRIVLI
+B_LAST_IN_INC -Dprefix=/opt/ActivePerl-5.8 -Dprivlib=/opt/ActivePerl-
+5.8/lib -Darchlib=/opt/ActivePerl-5.8/lib -Dsiteprefix=/opt/ActivePer
+l-5.8/site -Dsitelib=/opt/ActivePerl-5.8/site/lib -Dsitearch=/opt/Act
+ivePerl-5.8/site/lib -Dsed=/bin/sed -Dconfig_heavy=Config_static.pl -
+Dcf_by=ActiveState -Dcf_email=support@ActiveState.com'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemulti
+plicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS
+ -DUSE_SITECUSTOMIZE -DNO_HASH_SEED -DPRIVLIB_LAST_IN_INC -fno-strict
+-aliasing -pipe -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/inc
+lude/gdbm',
    optimize='-O2',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DUSE_SIT
+ECUSTOMIZE -DNO_HASH_SEED -DPRIVLIB_LAST_IN_INC -fno-strict-aliasing 
+-pipe -I/usr/include/gdbm'
    ccversion='', gccversion='2.96 20000731 (Red Hat Linux 7.1 2.96-85
+)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1
+2
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
+ lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =''
    libpth=/lib /usr/lib /usr/local/lib
    libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lpthread -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=/lib/libc-2.2.4.so, so=so, useshrplib=false, libperl=libperl.
+a
    gnulibc_version='2.2.4'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2'


Characteristics of this binary (from libperl):
  Compile-time options: MULTIPLICITY PERL_IMPLICIT_CONTEXT
                        PERL_MALLOC_WRAP THREADS_HAVE_PIDS USE_ITHREAD
+S
                        USE_LARGE_FILES USE_PERLIO USE_REENTRANT_API
                        USE_SITECUSTOMIZE
  Locally applied patches:
        ActivePerl Build 822 [280952]
        Iin_load_module moved for compatibility with build 806
        PerlEx support in CGI::Carp
        Less verbose ExtUtils::Install and Pod::Find
        Patch for CAN-2005-0448 from Debian with modifications
        Rearrange @INC so that 'site' is searched before 'perl'
        Partly reverted 24733 to preserve binary compatibility
        MAINT31223 plus additional changes
        31324 Fix DynaLoader::dl_findfile() to locate .so files again
        26970 Make Passive mode the default for Net::FTP
        24699 ICMP_UNREACHABLE handling in Net::Ping
  Built under linux
  Compiled at Jul 31 2007 20:53:37
  @INC:
    /opt/ActivePerl-5.8/site/lib
    /opt/ActivePerl-5.8/lib

[download]

In reply to Re^2: Memory allocation/performance issue for large strings (of binary data) under Windows. by Anonymous Monk
in thread Memory allocation/performance issue for large strings (of binary data) under Windows. by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.