It's hard to give specific advice without seeing a sample of your input data, but it may be hard for you to show us a sample if your lines are 550 columns wide. Short answer: yes, it's quite likely that there is a faster solution using split and/or regex than what you're doing now with multiple index and substr commands.

One example to get you started: If you want the six columns numbered 10, 20, 30, 32, 34, and 38 (with the first column numbered 0), and your columns are separated by whitespace, you could do this to get them in an array:

while(my $line = <$file>){
  chomp $line;
  # split the line on whitespace, then take certain indexed columns
  my @columns = (split /\s+/, $line)[10,20,30,32,34,38];
  do_stuff_with_those_columns(@columns);
}
[download]

On the other hand, if you know what you want out of each column, you may be able to skip the step of splitting into an array and go straight to extracting what you need. It just depends on the data. Give us at least a couple lines if you can.

I really can't give any data as it is customer sensitive. Assume I have to search 10 columns for key1= or key2= or key3= or key4=. There is no way to predict what will be after those 4 words. But I have to strip out whatever follows up to a & or reach end of the field. I have to blank the original data where it comes from (just what I am pulling, not whole column) and then append new columns with what I stripped out. I like the idea of the slice syntax. But then how would I go back and update those same columns?

> I really can't give any data as it is customer sensitive

please invent non-sensitiv data for your use case.

Cheers Rolf
_{(addicted to the Perl Programming Language and ☆☆☆☆ :)
Je suis Charlie!}

Can you make up some random but representative data?

For example, it sounds a lot like you're parsing a URL for GET parameters, so your data with fake customer data with might be:
http://www.example.com/random_funky_characters_here.html?name=JoeSmith&address=123UniversalAve&CreditCardNumber=1234567890&kneecapstate=broken

my $inrec='0;1;2?foo=duh;3;4;5;6;7;8;9;;;;;;';


You've already given some input data, presumably faked to the necessary degree. All you needed to do was to give a couple more fake input record instances and their corresponding output records to make the needed transformation(s) much more clear. No "customer sensitive" data need apply.

---

Give a man a fish:  <%-(-(-(-<

You can strip out the values while moving them. From what you describe, I'd ignore the columns (unless it's possible for those key2= patterns to appear in other columns where you want to ignore them). So you could use something like this to get the value for key1, strip it out, and append it to the end of the line:

open my $in,  '<', 'infile'  or die $!; # open input file
open my $out, '>', 'outfile' or die $!; # open output file
while(my $line = <$in>){
  chomp $line;                       # remove newline
  if( $line =~ s/key1=([^&\s]+)// ){ # capture value in $1 while 
                                     #   replacing with empty string
    $line .= " key1=$1";             # append key and value to line
  }
  print $out "$line\n";              # print line to output file
}
close $in;
close $out;
[download]

That will do one find/replace/copy, so you can repeat the if loop while changing the key?= to do multiple ones.

Aaron B.
Available for small or large Perl jobs and *nix system administration; see my home node.

sub not_so_special {
    my $s = shift;
    my @cols = split /;/, $s, -1;
    my $e;
    $cols[2] =~ s/=(.+)/'=' . (' ' x length($e = $1))/e;
    return join ';', @cols, $e
}
[download]

Update
Or, maybe, more readable and a tiny bit faster:

sub new {
    my @cols = split /;/, shift, -1;
    $cols[2] =~ s/=(.+)/=/;
    $cols[2] .= ' ' x length $1;
    return join ';', @cols, $1
}
[download]

Update 2
This seems even faster (but uglier):

use Syntax::Construct qw{ /r };
sub newer {
    shift =~ s/^([^;]*;[^;]*;)(.*?=)([^;]*)(;.*)/"$1$2" . (' ' x lengt
+h $3) . "$4;$3"/er
}
[download]

To grab several columns from split, you can use the slice syntax:

my @columns = (split /[?=]/)[3 .. 6, 9];
[download]

I'm a bit confused about just what you want to delete and what you want to keep, but here's an approach. I've no idea about its speed. It needs Perl version 5.10+. Note that field offsets are zero-based.

c:\@Work\Perl\monks>perl -wMstrict -le
"use 5.010;
 ;;
 my $inrec = '0;1;2?foo=DUH;3;4;5;6?bar=BLAB DAB;7;8;9;;;;;;';
 print qq{begin: '$inrec'};
 ;;
 my $sep   = ';';
 my $data  = qr{ [^\Q$sep\E] }xms;
 my $field = qr{ $data* \Q$sep\E }xms;
 ;;
 my %targets = qw(foo 2  bar 6);
 my @words =
   sort { $targets{$a} <=> $targets{$b} }
   keys %targets
   ;
 ;;
 for my $word (@words) {
   $inrec =~ s{ \A (?:$field){$targets{$word}}
                [^?]+ \? \Q$word\E = \K ($data*)
              }
              {}xms;
   $inrec .= qq{$1$sep};
   print qq{'$inrec'};
   }
 ;;
 print qq{final: '$inrec'};
"
begin: '0;1;2?foo=DUH;3;4;5;6?bar=BLAB DAB;7;8;9;;;;;;'
'0;1;2?foo=;3;4;5;6?bar=BLAB DAB;7;8;9;;;;;;DUH;'
'0;1;2?foo=;3;4;5;6?bar=;7;8;9;;;;;;DUH;BLAB DAB;'
final: '0;1;2?foo=;3;4;5;6?bar=;7;8;9;;;;;;DUH;BLAB DAB;'
[download]

Update: Changed code example to make more consistent use of the  $sep scalar.

(undef, $captured) = split(/ekey=(.+)/, $_);
[download]

I think I would rather explicitly extract whatever it was I was interested in rather than relying on a somewhat obscure feature of the split built-in:

c:\@Work\Perl\monks>perl -wMstrict -le
"$_ = qq{xxx ekey=foo bar \n xyzzy};
 ;;
 my (undef, $captured) = split(/ekey=(.+)/, $_);
 print qq{captured '$captured'};
 ;;
 my ($bagged) = $_ =~ m{ ekey= ([^\n]+) }xms;
 print qq{  bagged '$bagged'};
"
captured 'foo bar '
  bagged 'foo bar '
[download]

---

Give a man a fish:  <%-(-(-(-<