Using Storablefreeze() on a packed scalar makes no sense at all!

Freezing is only useful for encoding data structures into single scalars.

A scalar is just a large chunk of ram with some header information, freezing it will achieve nothing useful. So just pass the scalar.

And then again, if you are having to pack the numbers you are reading from the socket, then you must be reading them in as ascii. And if once you've passed them to another thread, you are going to process them as numbers with Perl, then you are going to have to unpack them again before you can do so. So why are you packing them?

You will save (a little) space by packing them, but not so much as to be particularly significant. And given the extra overhead (time) it takes to put them through the pack/unpack, and time seems to be your limitation, don't do that. You could just concatenate (or better; overwrite a buffer with) the ascii as you read it and then split it up when you've passed it to the processing thread.

The only real advantage of packing is that it allows you to know how much space is required to hold a given number of integers. But unless you know how many you are going to read and pass, that isn't much advantage. You will still need to either: pre-allocate space larger than you ultimately will ever need; or accept that you will sometimes need to grow the buffer.

And finally, "passing a (large) scalar between threads", ineveitably means copying it. Depending upon how you "pass it", possibly 2 or more times. Better to pre-allocate a shared buffer (or two); overwrite the data directly into that buffer; and then pass a simple flag between the threads to tell the processing thread which buffer contains the data that is ready for processing.

Your mocked up test script doesn't tell me enough about your real script to allow me to offer something representative:

• Do you need all 7 million integers before you can start processing?
• Will you be reading/passing/processing, just a single batch of 7 million numbers?
• Or many batches of 7 million numbers?
• Or is 7 million a guestimate of the total, that you will read in many smaller batches?
• How will the socket reader know when it has finished reading a batch?

You are correct, my example is rather flawed and in hindsight I could have simplified the example and removed the pack.

The packed structure eventually gets to C code in an XS module. Basically the Perl side is handling the parsing/socket logic and I am using freeze/thaw as a generic message mechanism to serialize perl objects to pass across threads (an event based threading model, sockets and GUI drive event). Some of these objects get 'thawed' in a pure C thread, while others end up in perl threads.

To Answer your direct questions:

Yes, I need all 7 million - it's a single transaction. Batch size is unknown. Sometimes the batch is small (a couple of thousand) to a couple of million. Unknown number of batches, sometimes 1, sometimes hundreds of batches. An end message is sent on socket to end a batch (the server is a 3rd party so I can't change the API).

In hindsight the question should have been "Why is growing a large scalar in windows so slow?"

My results from running the example in linux (perl -V below)

0.06,0.02
0.4,0.27
2.44,0.27
2.89,0.63
Finished
[download]

and commenting out the line that preallocates:

0.07,0.01
0.07,0.01
2.43,0.38
2.91,0.76
[download]

not much difference. The windows version is over a 100 times slower (at least with my machine and Perl). Why?

Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
  Platform:
    osname=linux, osvers=2.2.24-7.0.3, archname=i686-linux-thread-mult
+i
    uname='linux redhat-70-i386.activestate.com 2.2.24-7.0.3 #1 fri ma
+r 14 08:28:25 est 2003 i686 unknown '
    config_args='-ders -Dcc=gcc -Dusethreads -Duseithreads -Ud_sigsetj
+mp -Uinstallusrbinperl -Ulocincpth= -Uloclibpth= -Accflags=-DUSE_SITE
+CUSTOMIZE -Duselargefiles -Accflags=-DNO_HASH_SEED -Accflags=-DPRIVLI
+B_LAST_IN_INC -Dprefix=/opt/ActivePerl-5.8 -Dprivlib=/opt/ActivePerl-
+5.8/lib -Darchlib=/opt/ActivePerl-5.8/lib -Dsiteprefix=/opt/ActivePer
+l-5.8/site -Dsitelib=/opt/ActivePerl-5.8/site/lib -Dsitearch=/opt/Act
+ivePerl-5.8/site/lib -Dsed=/bin/sed -Dconfig_heavy=Config_static.pl -
+Dcf_by=ActiveState -Dcf_email=support@ActiveState.com'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemulti
+plicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS
+ -DUSE_SITECUSTOMIZE -DNO_HASH_SEED -DPRIVLIB_LAST_IN_INC -fno-strict
+-aliasing -pipe -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/inc
+lude/gdbm',
    optimize='-O2',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DUSE_SIT
+ECUSTOMIZE -DNO_HASH_SEED -DPRIVLIB_LAST_IN_INC -fno-strict-aliasing 
+-pipe -I/usr/include/gdbm'
    ccversion='', gccversion='2.96 20000731 (Red Hat Linux 7.1 2.96-85
+)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1
+2
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
+ lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =''
    libpth=/lib /usr/lib /usr/local/lib
    libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lpthread -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=/lib/libc-2.2.4.so, so=so, useshrplib=false, libperl=libperl.
+a
    gnulibc_version='2.2.4'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2'


Characteristics of this binary (from libperl):
  Compile-time options: MULTIPLICITY PERL_IMPLICIT_CONTEXT
                        PERL_MALLOC_WRAP THREADS_HAVE_PIDS USE_ITHREAD
+S
                        USE_LARGE_FILES USE_PERLIO USE_REENTRANT_API
                        USE_SITECUSTOMIZE
  Locally applied patches:
        ActivePerl Build 822 [280952]
        Iin_load_module moved for compatibility with build 806
        PerlEx support in CGI::Carp
        Less verbose ExtUtils::Install and Pod::Find
        Patch for CAN-2005-0448 from Debian with modifications
        Rearrange @INC so that 'site' is searched before 'perl'
        Partly reverted 24733 to preserve binary compatibility
        MAINT31223 plus additional changes
        31324 Fix DynaLoader::dl_findfile() to locate .so files again
        26970 Make Passive mode the default for Net::FTP
        24699 ICMP_UNREACHABLE handling in Net::Ping
  Built under linux
  Compiled at Jul 31 2007 20:53:37
  @INC:
    /opt/ActivePerl-5.8/site/lib
    /opt/ActivePerl-5.8/lib
[download]

FWIW: I can confirm that I see similar extreme differences between Ubunto/5.10.0 & Vista/5.10.1 :(

Looking at the differences in compile-time options used:

Compile-time options: 

Linux:
THREADS_HAVE_PIDS 
USE_REENTRANT_API 


Windows:
PERL_IMPLICIT_SYS 
PL_OP_SLAB_ALLOC 
USE_FAST_STDIO
[download]

Which, if any, of those is the source of the difference ("SLAB_ALLOC"?) will require someone with a much better understanding of the internals to answer.

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

RIP PCW It is as I've been saying!(Audio until 20090817)

So precreate the string. Doing so doesn't cost a lot (assuming your sample represents your real code) so probably doesn't impact performance much for small numbers of iterations but, as you have discovered, provides a big win for large numbers of iterations.

If you really don't want to do that, you can get most of the benefit by using a piecemeal preallocate and copy technique:

use strict;
use warnings;

#number of items to pack
my $iter = 1000000;
my $string;
my $startAlloc = 0;
my $preAllocSize = $startAlloc;

Time ();
$$string = '';

#now lets create the data block...
for my $count (1 .. $iter) {
    if ($preAllocSize && length ($$string) >= $preAllocSize) {
        $preAllocSize = length ($$string) * 2;
        my $newStr = 1 x $preAllocSize;
        $newStr = $$string;
        $string = \$newStr;
    }

    $$string .= 'x' x 20;
}
Time ();
print "Finished with start alloc: $startAlloc\n";
print "Final string length: ", length ($$string);

sub Time {
    my ($user, $system, $cuser, $csystem) = times;
    print "$user,$system\n";
}
[download]

for two different runs prints:

0.015,0
18.345,29.78
Finished with start alloc: 0
Final string length: 20000000

0.015,0
0.374,0.046
Finished with start alloc: 1
Final string length: 20000000
[download]