The problem seems to be pretty definitely sourced within the MS CRT.

If I compile and run this using gcc under Ubuntu running inside a VirtualBox emulator (which ought to carry some overheads):

#include <stdio.h>
#include <stdlib.h>
#include <time.h>

int main( int argc, char** argv ) {
    long long i, n = 1000000;
    char *p = (char*)malloc( 1000 );
    time_t start, finish;
    double elapsed;

        time( &start );
    for( i = 2; i < n; ++i ) {
//        printf( "\r %lld\t", i * 1000 );
        if( ! ( p = (char*)realloc( p, 1000 * i ) ) ) {
            printf( "\nFailed to realloc to %lld\n", i * 1000 );
            exit( 1 );
        }
    }
    time( &finish );
    elapsed = difftime( finish, start );

    printf( "\nfinal size: %lld; took %.3f seconds\n", n * 1000, elaps
+ed );
    exit( 0 );
}
[download]

It takes less that a second to realloc a buffer to 1GB in 1000 byte increments:

mehere@mehere-desktop:~$ gcc mem.c -o memtest
mehere@mehere-desktop:~$ ./memtest 

final size: 1000000000; took 0.000 seconds
[download]

However, if I compile and run this using MS VC++:

#include <stdio.h>
#include <time.h>

int main( int argc, char** argv ) {
    __int64 i, n = 1000000;
    char *p = (char*)malloc( 1000 );
    time_t start, finish;
    double elapsed;

    time( &start );
    for( i = 2; i < n; ++i ) {
//        printf( "\r %I64d\t", i * 1000 );
        if( ! ( p = (char*)realloc( p, i * 1000 ) ) ) {
            printf( "\nFailed to realloc to %I64d\n", i * 1000 );
            exit( 1 );
        }
    }
    time( &finish );
    elapsed = difftime( finish, start );

    printf( "\nfinal size: %I64d; took %.3f seconds\n", n * 1000, elap
+sed );
    exit( 0 );
}
[download]

it takes over an hour to run. I haven't had the patience to let it complete yet!

I also compiled it with MinGW (which appears to also use the MSCRT?), and it has taken 1hr 20 mins (so far and appears to only 1/4 of the way there).

The problem seems to lie with the CRT realloc() which grows the heap in iddy-biddy chunks each time.

In addition, it may be

• walking the heap attempting to coallesce freespace prior to allocating extra virtual memory.
• The virtual memory allocator zeros new commits prior to copying over teh existing data into the reallocated space.

Bottom line: The MSCRT heap management routines are crap!