in reply to search/replace very large file w/o linebreaks

One thing you could try is to replace the string Perl uses to identify the ends of lines (input record separator) with one of the tags in your file.

{
  local $/ = '<tag>';
  while (<>) {
    # do stuff
  }
}

[download]

Or you could just feed a while loop chunks of data at a time using a trick described in perldoc perlvar:

   Setting $/ to a reference to an integer, scalar 
   containing an integer, or scalar that's convertible to an
   integer will attempt to read records instead of lines, 
   with the maximum record size being the referenced
   integer.  So this:

   local $/ = \32768; # or \"32768", or \$var_containing_32768
   open my $fh, $myfile or die $!;
   local $_ = <$fh>;

   will read a record of no more than 32768 bytes from FILE.  
   If you're not reading from a record-oriented
   file (or your OS doesn't have record-oriented files),    
   then you'll likely get a full chunk of data with
   every read.  If a record is larger than the record size 
   you've set, you'll get the record back in pieces.

[download]

Update: Commented out irrelevant text from perlvar.

--
Allolex

Replies are listed 'Best First'.
Re: Re: search/replace very large file w/o linebreaks
by ysth (Canon) on Jan 08, 2004 at 17:39 UTC
    If one of the tags overlaps the record boundary with the fixed length read, you can do the substitition on two blocks at a time. Pseudocode: 
    use constant BLOCKLENGTH => 32768; # > length of any tag
$\ = \BLOCKLENGTH;
my $buffer = '';
while (!eof()) {
   $buffer .= <>;
   s/tag1/\n/g;
   s/tag2/\t/g;
   # leave BLOCKLENGTH chars in $buffer,
   # print however much comes before that
   print substr($buffer, 0, -BLOCKLENGTH, '');
};
# print whatever's left
print $buffer;

    [download]
    Alternatively, just do a while(<>) loop with the substitution and print inside, but run it twice with two different record lengths such that no multiples (less than the file size) of the two come within a distance of each other less than the length of the longest tag. Don't know quite how to go about calculating such numbers, though. Any number theoreticians here?
[reply]
[d/l]
[select]
      if you're reading in chunks you might aswell use sysread: 
      sub BLOCKLENGTH () { 1 << 12 }; # TIMTOWTDI =)

$_ = '';
while(sysread STDIN, $_, BLOCKLENGTH, length){
     s///g; # you know
     syswrite STDOUT, substr($_, 0, -BLOCKLENGTH, ''); # the fourth ar
+gument to substr will replace 0 .. -BLOCKLENGTH
};

syswrite STDOUT, $_;

      [download]
      It is more suited for the task, takes a bit less memory, and might even be faster if your stdio is stoned.

      Personally, i think that perl -pe 'BEGIN{ $\ = "\n"; $/ = "tag" } chomp; s/tag2/\t/g; print' < infile > outfile is the nicest way.

       

      Update: I thought some explanation was appropriate.

      The notion of what is a line is pretty flexible, and has to be (computers in general and specifically in perl). A line, traditionally, ended in a carrige return and a line feed, in one order or another. Windoze still uses that. MacOS uses only CRs, ~UNIX does only LF (i might be confused). The one byte solution is somwhat simpler. But since you need to support two bytes in case they come, why not support everything.

      Enters the concept of a record.

      Treating a line as a record, with either a fixed length ($\ = \ 123), or one ending with a certain string ($\ = "\n" is for a record which is also a line on your native system) adds the flexibility to do something like you wanted quite easily. You're translating a record format that ends in a certain string, to one that ends with newlines. $\ is the output record seperator, BTW.

      -nuffin
      zz zZ Z Z #!perl
[reply]
[d/l]
[select]
        But the sysread example sucks whenever a tag is inbetween your BLOCKLENGTH.

        Update:I was wrong and promise to read the snippsets more carefully. Want some XP? Ok!

        --
        Boris
[reply]
[d/l]

In Section Seekers of Perl Wisdom