-- Randal L. Schwartz, Perl hacker
Be sure to read my standard disclaimer if this is a reply.

I have been thinking about this idea, and I can make it work by doing something like this:

use strict;
open INFILE,"infile.txt";
open OFILE,">outfile.txt";
my $total_os=0;
while(<INFILE>){
  my $tmp=$_;
  $str=~s/(\s+)/osmarker(pos($str),$1)/xeg;

  # a bunch of regular expressions

  $total_os+=length($_);
  print OFILE $str;
}

sub osmarker{
  my $os=shift;
  my $spaces=shift;
  $os+=length($spaces)+total_os;
  return $spaces."<OS=$os>";
}
[download]

The problem that inserting this markers has is not on the data mining tool, but in the regular expressions that munge in the text. There are some that look for "WORD\s+WORD" that would be screwed up by this marker. I could fix this by defining some variable like this:

my $space=qr/(?:<OS=\d+>|\s)/;
[download]

and replacing all instances of "\s" with "$space". Is there an easier way of doing this? Is there a way to overload "\s"?

Keeping keep the markup in a separate file is called parallel markup. With this approach the original document remains intact.

Even if you are as big a fan of markup as I am, it is interesting to see a different way to do it.

It should work perfectly the first time! - toma

Without knowing the nature of the substitutions you are making, or what the original file is, or how it is used, it's difficult to know whether this idea might fly or not, but here it is anyway.

When I first started coding--in assembler, many moons ago--it was quite common practice to embed blocks of 16 or 32 nop's at the end of each block of code.

These blocks of nops where called "patch areas". The idea being that if once compiled, a bug was found in a program, it was possible to patch the executable rather than having to re-build it, and these patch areas allowed for the potential that a routine needed to grow in size.

Why would you do this reather than re-build. Well, it was consider safer to patch the executable as there is considerably less likelyhood of unwittingly making other changes. Eg. Someone omits or adds a compiler switch, #define or whatever, and having fixed one bug, you suddenly start getting several others show up in completely unrelated parts of the code. I believe that the technique is still actively used in such things as satallite control software, the space shuttle etc.

Anyway, back to the question. Depending upon the nature of the substitutions you are making, you might be able to adjust the substitutions such that (most of) the offsets within the file remain the same after substitution as before. For net deletions this is fairly easy. If the replacement text is shorter than the original, you can pad it--with spaces or nulls for example.

The problem comes when the replacement is longer than the original. Depending upon the nature of the original file, and what applications are used to view/manipulate it, you might get away with adding some 'patch space' to it.

For instance: if you added 10 or 20 spaces or null bytes to the end of each line, it might give you latitude to make the substitutions and have enough play to adjust the padding at the end of the following or previous lines to compensate for the changes. Obviously this wouldn't by itself cater for all possibilities. You might need to add a few nulls to the end of each word in the original file.

Having typed all that, I think that the effort involved in getting the padding juggling algorithm correct would probably be much more than building a lookup table to do the mapping, but there you go. Only you will know if this has any merit for your situation.

Run the same regexes over the original file only dont
substitute  --> then save the positions ....

Then make another run over the original file, this
time substitute and write into the new file.

-bl0rf