in reply to Re^3: Out of memory problems
in thread Out of memory problems

Thanks for that insight. I am working on a byte boundary. If I know exactly the bytes I need to keep and the ones I need to throw away is there a simpler method. I will need to extract the same byte positions in every 384 bytes. Using a substituation was the only way I knew how to do it but sounds like you have alot more experience with this. I tried to edit the file using hex in the files original state (which is packed binary on a sun machine running Solaris 5.6) but for every hex character I removed 2 bytes would be extracted. Any thoughts on this. And thanks for the help.

Replies are listed 'Best First'.
Re^5: Out of memory problems
by BrowserUk (Patriarch) on Oct 21, 2004 at 22:13 UTC

    Okay, knowing that the bit patterns are byte aligned and the records are 384 bytes each makes thing much easier. The following program should do the trick quite efficiently.

    #! perl -slw use strict; use bytes; ## Force byte semantic rather than (unicode) character sema +ntics. ## Maybe unnecessary, but it won't hurt. ## Using ':raw' is (roughly) equivalent to using [binmode] open IN, '< :raw', $ARGV[ 0 ] or die "$ARGV[ 0 ] : $!"; open OUT, '> :raw', $ARGV[ 1 ] or die "$ARGV[ 1 ] : $!"; local $/ = \384; ## Read file in 384 byte chunks. ## The only magic here is that ## a) we are reading the file in 384 byte chunks ## b) The unpack format extracts just the bits we wish to retain ## from the exact offsets we need. ## x2 - skips the xF4 x?? bytes ## A190 - returns the 190 bytes we need to retain ## x2 - skips the second xF4 x?? bytes ## A58 - returns the next 58 bytes. ## - the remaining 132 bytes from the buffer are discar +ded. while( <IN> ) { print OUT unpack 'x2 A190 x2 A58', $_; } close IN; close OUT;

    In theory, the following 'one-liner' (I've wrapped it to save PM the bother) would also work.

    If you have the time/patience/enthusiasm to try it, I'd love to know if it does.

    You will have to twiddle the "s -v- 's issue if you're on a non-Win32 platform.

    perl -C0 -mbytes -e"BEGIN{$/=\384}" -ne"print unpack 'x2 A190 x2 A58', $_" <infile >outfile

    Examine what is said, not who speaks.
    "Efficiency is intelligent laziness." -David Dunham
    "Think for yourself!" - Abigail
    "Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon