Does there happen to be more than one line in each file. If so then print "$file\n" will be executed once per line. Try moving that line outside the while loop to see how many times the files are called.

Second, since I am not completely sure of what you want. My guess is that you want each files input independent of the other. (in other words you want the variables inside the while loop initialize before you do anything with them. My guess (and only a guess) is you want something like the following (not tested):

use strict;
use warnings;

my @files = grep /^output\d+$/, <output*>;
foreach my $file ( @files ) {
  open (IN, "<$file") 
      or die "could not open $file: $!";
  print "$file\n";
  my ($total,$i) = (0,0); #initialize to zero for each file
  while ( my $input = <IN> ){
    #sine you don't seem to be using $x, $y, $z anyhow
    my $sum = (split /\s+/,$input)[3]; 
    $total += $sum;
    $i++;
    my $div = $total/$i;
    my $sqrt = sqrt($div);   
    print "$total\n";
    open (OUT2, ">RMSD$file") 
        or die "could not open RMSD$file: $!";    
    print OUT2 "$sqrt\n"
  }
}
[download]

HTH

-enlil