Re: Memory use/leak with large number of (?{}) patterns in regex

Well, the lack of freeing is unremarkable. Running the regex is likely to malloc() and and finally free() lots of small chunks of memory. These will be reused if you run a similar regex again, but trying to then malloc() a single 1Gb string is unlikely to be able to make use of all those little blocks recently freed.

However, what *is* worrying is that memory usage goes quadratic on the number of code blocks in the pattern. I'll try to have a look at it sometime when I have the time.

Dave.

Comment on Re: Memory use/leak with large number of (?{}) patterns in regex

Replies are listed 'Best First'.
Re^2: Memory use/leak with large number of (?{}) patterns in regex by dave_the_m (Monsignor) on Nov 24, 2019 at 20:00 UTC
It's the combination of captures and code blocks. Each time the regex engine is about to execute a code block, it saves the indices of all the captures done so far, so they can be restored at the end. It does this on the pessimistic assumption that code within the block can do anything, including recursively executing the same regex again, overwriting the existing capture indices. This is why quadratic memory behaviour is being seen. Not ideal, but can avoided if you use non-capturing braces. Dave.	[reply] [d/l]
Re^3: Memory use/leak with large number of (?{}) patterns in regex by jcb (Parson) on Nov 25, 2019 at 01:07 UTC
Could saving the capture indices be lazily done, with some kind of "regex in use" flag set on the regex, such that recursively executing the same regex causes the capture indices to be preserved, but only if really needed? This would slightly add to the general regex overhead, from needing to check the "regex in use" flag on every pattern match, but perhaps that could be folded into the existing logic that handles compiling patterns when needed?	[reply]
Re^4: Memory use/leak with large number of (?{}) patterns in regex by dave_the_m (Monsignor) on Nov 25, 2019 at 09:15 UTC
There's a lot that could be done. It's a complete mess at the moment and needs an overhaul - it affects lots of things, not just this issue, e.g. unnecessary slowness using a regex object for a match compared with a literal pattern. Really, the capture state needs splitting off into a separate data structure from the main regex data structure, so that it can be swapped in and out easily, and so a qr// object can be used in multiple places without internally having to clone the whole thing each time. It's on my very long list of things to be do, but I'm not likely to do it any time soon. Dave.	[reply]