Hmm. I'm not surprised your processing is slow. Converting the whole 3 GB to ascii-ized binary will produce a 24 GB file. Then search/replace on that can convert back as separate stages?

Looking at your regex, it looks like your patterns are probably byte aligned? Very few processes produce data that is not...but I realise that it is possible. This how would tackle the task assuming tha the data is byte aligned.

The technique is called a 'sliding buffer'. You read a buffer of at least double the size of the thing you are looking for. In this case, 384 x 2 = 768 bytes. You then apply the regex to the whole of that buffer, write out the first half of it, move the second half to the front, top it up from the input file and re-apply the regex to the whole thing.

Note. That is a simplified description and the following code implements that simplified description by way of example only. It isn't tested and almost certainly won't work as is.

#!/usr/local/bin/perl -w
use strict;

open IN,  '< :raw', $ARGV[ 0 ] or die "$ARGV[ 0 ] : $!";
open OUT, '> :raw', $ARGV[ 1 ] or die "$ARGV[ 1 ] : $!";

my $buffer;
sysread IN, $buffer, 384, 384;
do{
    ## Move the second half of the buffer to the front.
    $buffer = substr( $buffer, 384 );
    
    ## and overwrite it with a new chunk
    sysread IN, $buffer, 384, length( $buffer );    
    
     ## Apply the regex
     $buffer =~ s[
         \xF4 .        ## The marker byte plus friend
         ( .{190} )  ## 1520 bits to retain
         \xF4 .        ## Second marker + friend
         ( .{58} )    ## 464 bits to retain
         .{132}        ## 1056 bits to discard
     ][$1$2]xg;
     
     ## Write out the first half of the buffer
     syswrite OUT, $buffer, 384;
} until eof IN;

close IN;
close OUT;
[download]

The are some further complications that need to be addressed.

• If your data is not byte aligned, then you can still unpack/pack the binary data on input/output and use your original regex--but avoid this if it is not necessary as it will slow your process down by an order of magnitude a lot.
• As you are modifying the buffer when the regex matches, you will need to apply different handling when a match has occured, and when it has not.

  When no substitution has taken place, the above method should be okay.

  When a substitution has occured, you will need to ensure that you do not re-process that amount of the buffer upto the place where the end of the last substitution occured--other wise you could get a false match/substitution occuring. See perlfunc:pos and /or perlvar @- & @+ for more information.

  Basically, you need to find out which part of the buffer has already been processed and write that out and only copy the remainder to the front of the buffer before refilling it to the x2 length.

  It will take thought, and testing (and testdata) to get this right! It is non-trivial to do and harder to describe.

• Using a larger buffer.

  Once you have the process working, instead of using a x2 buffer, you could save considerable time by reading and writing much larger chunks-- 1, 2, 10, or even 100 MB would be possible.

  The only thing to remember is that if no match occurs you must retain the last 384 bytes from the buffer and place these at the front of the next to ensure full coverage.

  If one (or more) substitutions have occurred, then you must retain everything from the end of the last substitution to the end--if that less than 384 bytes--or the last 384 bytes.

Processing your 3 GB this way should take no more than 1/2 hour if you can avoid the packing/unpacking. And not much more than a couple of hours if you cannot. These are ballpark figures!

I hope that rather sketchy explaination makes some kind of sense. Good luck.

Thanks for that insight. I am working on a byte boundary. If I know exactly the bytes I need to keep and the ones I need to throw away is there a simpler method. I will need to extract the same byte positions in every 384 bytes. Using a substituation was the only way I knew how to do it but sounds like you have alot more experience with this. I tried to edit the file using hex in the files original state (which is packed binary on a sun machine running Solaris 5.6) but for every hex character I removed 2 bytes would be extracted. Any thoughts on this. And thanks for the help.

I tried the code you mentioned with a few tweeks but am not being successful. The first 4 frames are good but after the patterns I want to remove are still there. I don't know why it would only appear to work for 4 frames and then quit. Any ideas??