Re: Memory allocation/performance issue for large strings (of binary data) under Windows.

Using Storablefreeze() on a packed scalar makes no sense at all!

Freezing is only useful for encoding data structures into single scalars.

A scalar is just a large chunk of ram with some header information, freezing it will achieve nothing useful. So just pass the scalar.

And then again, if you are having to pack the numbers you are reading from the socket, then you must be reading them in as ascii. And if once you've passed them to another thread, you are going to process them as numbers with Perl, then you are going to have to unpack them again before you can do so. So why are you packing them?

You will save (a little) space by packing them, but not so much as to be particularly significant. And given the extra overhead (time) it takes to put them through the pack/unpack, and time seems to be your limitation, don't do that. You could just concatenate (or better; overwrite a buffer with) the ascii as you read it and then split it up when you've passed it to the processing thread.

The only real advantage of packing is that it allows you to know how much space is required to hold a given number of integers. But unless you know how many you are going to read and pass, that isn't much advantage. You will still need to either: pre-allocate space larger than you ultimately will ever need; or accept that you will sometimes need to grow the buffer.

And finally, "passing a (large) scalar between threads", ineveitably means copying it. Depending upon how you "pass it", possibly 2 or more times. Better to pre-allocate a shared buffer (or two); overwrite the data directly into that buffer; and then pass a simple flag between the threads to tell the processing thread which buffer contains the data that is ready for processing.

Your mocked up test script doesn't tell me enough about your real script to allow me to offer something representative:

Do you need all 7 million integers before you can start processing?
Will you be reading/passing/processing, just a single batch of 7 million numbers?
Or many batches of 7 million numbers?
Or is 7 million a guestimate of the total, that you will read in many smaller batches?
How will the socket reader know when it has finished reading a batch?

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

RIP PCW It is as I've been saying!(Audio until 20090817)

Comment on Re: Memory allocation/performance issue for large strings (of binary data) under Windows. Download Code

Replies are listed 'Best First'.
Re^2: Memory allocation/performance issue for large strings (of binary data) under Windows. by Anonymous Monk on Nov 30, 2009 at 06:10 UTC
You are correct, my example is rather flawed and in hindsight I could have simplified the example and removed the pack. The packed structure eventually gets to C code in an XS module. Basically the Perl side is handling the parsing/socket logic and I am using freeze/thaw as a generic message mechanism to serialize perl objects to pass across threads (an event based threading model, sockets and GUI drive event). Some of these objects get 'thawed' in a pure C thread, while others end up in perl threads. To Answer your direct questions: Yes, I need all 7 million - it's a single transaction. Batch size is unknown. Sometimes the batch is small (a couple of thousand) to a couple of million. Unknown number of batches, sometimes 1, sometimes hundreds of batches. An end message is sent on socket to end a batch (the server is a 3rd party so I can't change the API). In hindsight the question should have been "Why is growing a large scalar in windows so slow?" My results from running the example in linux (perl -V below) `0.06,0.02 0.4,0.27 2.44,0.27 2.89,0.63 Finished` [download] and commenting out the line that preallocates: `0.07,0.01 0.07,0.01 2.43,0.38 2.91,0.76` [download] not much difference. The windows version is over a 100 times slower (at least with my machine and Perl). Why? Summary of my perl5 (revision 5 version 8 subversion 8) configuration: Platform: osname=linux, osvers=2.2.24-7.0.3, archname=i686-linux-thread-mult +i uname='linux redhat-70-i386.activestate.com 2.2.24-7.0.3 #1 fri ma +r 14 08:28:25 est 2003 i686 unknown ' config_args='-ders -Dcc=gcc -Dusethreads -Duseithreads -Ud_sigsetj +mp -Uinstallusrbinperl -Ulocincpth= -Uloclibpth= -Accflags=-DUSE_SITE +CUSTOMIZE -Duselargefiles -Accflags=-DNO_HASH_SEED -Accflags=-DPRIVLI +B_LAST_IN_INC -Dprefix=/opt/ActivePerl-5.8 -Dprivlib=/opt/ActivePerl- +5.8/lib -Darchlib=/opt/ActivePerl-5.8/lib -Dsiteprefix=/opt/ActivePer +l-5.8/site -Dsitelib=/opt/ActivePerl-5.8/site/lib -Dsitearch=/opt/Act +ivePerl-5.8/site/lib -Dsed=/bin/sed -Dconfig_heavy=Config_static.pl - +Dcf_by=ActiveState -Dcf_email=support@ActiveState.com' hint=recommended, useposix=true, d_sigaction=define usethreads=define use5005threads=undef useithreads=define usemulti +plicity=define useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS + -DUSE_SITECUSTOMIZE -DNO_HASH_SEED -DPRIVLIB_LAST_IN_INC -fno-strict +-aliasing -pipe -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/inc +lude/gdbm', optimize='-O2', cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DUSE_SIT +ECUSTOMIZE -DNO_HASH_SEED -DPRIVLIB_LAST_IN_INC -fno-strict-aliasing +-pipe -I/usr/include/gdbm' ccversion='', gccversion='2.96 20000731 (Red Hat Linux 7.1 2.96-85 +)', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1 +2 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', + lseeksize=8 alignbytes=4, prototype=define Linker and Libraries: ld='gcc', ldflags ='' libpth=/lib /usr/lib /usr/local/lib libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lpthread -lc perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc libc=/lib/libc-2.2.4.so, so=so, useshrplib=false, libperl=libperl. +a gnulibc_version='2.2.4' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -O2' Characteristics of this binary (from libperl): Compile-time options: MULTIPLICITY PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP THREADS_HAVE_PIDS USE_ITHREAD +S USE_LARGE_FILES USE_PERLIO USE_REENTRANT_API USE_SITECUSTOMIZE Locally applied patches: ActivePerl Build 822 [280952] Iin_load_module moved for compatibility with build 806 PerlEx support in CGI::Carp Less verbose ExtUtils::Install and Pod::Find Patch for CAN-2005-0448 from Debian with modifications Rearrange @INC so that 'site' is searched before 'perl' Partly reverted 24733 to preserve binary compatibility MAINT31223 plus additional changes 31324 Fix DynaLoader::dl_findfile() to locate .so files again 26970 Make Passive mode the default for Net::FTP 24699 ICMP_UNREACHABLE handling in Net::Ping Built under linux Compiled at Jul 31 2007 20:53:37 @INC: /opt/ActivePerl-5.8/site/lib /opt/ActivePerl-5.8/lib [download]	[reply] [d/l] [select]
Re^3: Memory allocation/performance issue for large strings (of binary data) under Windows. by BrowserUk (Patriarch) on Nov 30, 2009 at 08:38 UTC
FWIW: I can confirm that I see similar extreme differences between Ubunto/5.10.0 & Vista/5.10.1 :( Looking at the differences in compile-time options used: `Compile-time options: Linux: THREADS_HAVE_PIDS USE_REENTRANT_API Windows: PERL_IMPLICIT_SYS PL_OP_SLAB_ALLOC USE_FAST_STDIO` [download] Which, if any, of those is the source of the difference ("SLAB_ALLOC"?) will require someone with a much better understanding of the internals to answer. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. RIP PCW It is as I've been saying!(Audio until 20090817)	[reply] [d/l]