++Ratazong (when the Vote Fairy next visits) for this sliding window solution. But I have two quibbles:
Say the search string is 230 characters long. Then since each input line is 80 characters, the search string is 3 lines long (because 3 x 80 = 240 is the smallest multiple of 80 to be >= 230). So n is 3. But the pattern may begin near the end of an input line and stretch over 4 lines. So the minimum size of the sliding window is n + 1 (320 characters for the example search string).
Setting the window size to n + 1 lines will produce the smallest memory footprint. But it will also entail a large amount of processing, much of it duplicated, as the regex engine searches over and over within the same overlapping text. If the window size is, say, ten times the minimum (i.e., 3200 characters for the 230 character search string), only 3 of the ten lines need be duplicated in each subsequent window — already a significant saving in processing time. Determining an optimum window size — one which successfully balances memory usage against processing time — will depend on the OP’s requirements and available memory, and will likely require some trial-and-error. But I expect the savings in processing time will more than compensate for the time spent in optimising the window size.
Hope that helps,
| Athanasius <°(((>< contra mundum | Iustus alius egestas vitae, eros Piratica, |
In reply to Re^2: Out of memory
by Athanasius
in thread Out of memory
by sandy105
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |