in reply to Re^2: pack/unpack binary editing
in thread pack/unpack binary editing

Hmm. Then you have a problem that will require a little more effort. How will you determine which bits to throw away?

If you can do it whilst moving through the file in a single direction, or if you can construct a set of "editing instruction" ("delete bit3/byte 700004", "insert '010' at bit6 of byte 3002" etc.), whilst treating the file read-only, then sort those into byte/bit sequence.

You can then do the editing in a second linear pass through the file. You would keep a running buffer ( 0 - 7 bits ) of any odd bits. Appending those to the front of each buffer as you read it in, make any modification to that chunk of bits and then write int( bits--in-memory/8) bytes back out, retaining the leftover bits. Rince and repeat till done.

The problem with that is that when you re-order the editing instructions, you will need to acount for any shifts in byte/bit positions in order to account for teh effects of editing that will be done by earlier sequences. Not a hugely onorous task, but one that would need thourough testing on small files before you starting screwing with the big one.

It really depends on your answer to the question I posed first. How will the sequence of edits be determined. The answer to that will define the best strategy.

Examine what is said, not who speaks.
Silence betokens consent.
Love the truth but pardon error.

Replies are listed 'Best First'.
Re^4: pack/unpack binary editing
by tperdue (Sexton) on Feb 08, 2005 at 14:22 UTC
    I have a frame of 480 bytes in which I have a sync word of 10 bits long. I need to be able to find the start of my framing which may not occur on a byte boundary. I also have to take into consideration that the file may skew occationally, so I'd have to check frame by frame for my sync word. I've done this successfully using pack and unpack but it takes forever. I would like to slide through the file without having to pack or unpack.
[reply]
      I am in awe of BrowserUK's (and your) willngness to tackle this as a challenge.

      May I humbly offer a simpler suggestion? Do you have control of the file-generator's output? Can you -- by accepting a somewhat larger datafile -- align your data on 512-byte boundaries with padding? Then, the frame is always recognizable and there's room for an individual frame to grow or shrink a bit.
[reply]

      How long is your current attempt taking?

      The following code processes a 100MB file (including finding and recordng 25million hits of a 10 bit pattern) in ~ 20 seconds, and a 1GB file (250 million hits) in ~ 3 minutes 20 seconds.

      I make that a round 1/2 hour to process your 9GB. And probably much less as your hits will be less frequent and you can advance the buffer pointer by 480 bytes after each hit.

      It uses a basic sliding buffer to process the file in 1 MB chunks with an overlap of enough bytes to ensure continuity. (You'll need to verify the math of the byte/bit offset calculations).

      Code + some timings

      Read more... (2 kB)
      Examine what is said, not who speaks.
      Silence betokens consent.
      Love the truth but pardon error.
[reply]
[d/l]
        Sorry for not getting back sooner. I was out of the office and was doing everything from memory. Here is the code I'm using and for a 1 gig file this takes a few hours. I forgot that I have to change every 2 bits to a corresponding 2 bit sequence. I'm also only looking for the first sync pattern once. I was hoping to do this without unpacking and later packing the data. That's the biggest drawback. 
        die "USAGE: $0 input\n" if scalar(@ARGV) < 1;

$| = 1;
$/ = \960;

open IN, "$ARGV[0]" or die "$ARGV[0] : $!";
open OUT, ">tmp2";

my $y = 0;
my $value = '';

while (<IN>) {
     my $bits = unpack("b*", $_);
     @array = split(//, $bits);
     foreach $value (@array) {
          $y++;
          $tmp = $tmp . $value;
          if ($y == 2) {
               if ($tmp =~ /00/) {
                    $tmp = '11';
               } elsif ($tmp =~ /11/) {
                    $tmp = '00';
               }
               print OUT $tmp;
               $y = 0;
               $tmp = '';
          }
     }
}

close IN;
close OUT;

$/ = \7680;
my $x = 0;

#I LEFT THE OUTPUT AS ASCII-IZED BINARY AT THIS POINT
#THUS THE LARGE INCREASE IN FILE SIZE

open IN, "$tmp2";
open OUT, >$tmp3";

while (<IN>) {
     $_ =~ s/^.*(11111011000010001111011100010000)/$1/ if $x == 0;
     $x = 1;
     print OUT pack("b*", $_);
}

close IN;
close OUT;

        [download]

        10-02-2005 Janitored by Arunbear - added code tags, as per Monastery guidelines

[reply]
[d/l]

In Section Seekers of Perl Wisdom