An example of where the "Substitution loop" error is emitted from pp_hot.c:

    I32 maxiters;

...

    strend = s + len;
    slen = RX_MATCH_UTF8(rx) ? utf8_length((U8*)s, (U8*)strend) : len;
    maxiters = 2 * slen + 10;   /* We can match twice at each
                                   position, once with zero-length,
                                   second time with non-zero. */


...

                if (iters++ > maxiters)
                    DIE(aTHX_ "Substitution loop");
[download]

So, maxiters is a 32bit signed integer, set to twice the length of the string (which will be more than the number of characters if there are UTF8 characters in the string) plus a few.

The maximum value of an I32 is +2,147,483,647, so when your string gets to a little over 1GB in length, the maxiter variable wraps and you get nonsense.