That's worked around by scanning each piece of the file twice (except for the very beginning and end), once with more data before it and once with more data after it.

If this is a run-once-only problem, trying using a tag as the input record separator is the way to go, and only if that fails with out-of-memory go to another approach.

One other way would be to pick a common character or string from the file that won't appear in any tag (or appears at the end of any tags it is in). For instance, ' ', '>', or 'the' might work, given appropriate data.