Something that does not solve the issue, but makes it less painfull, it to replace the "push" with "unshift".
It causes the high values of $time to be used first, so there's one first large memory allocation rather than many extensions on the existing amount.