I find that the best XS code (easiest to understand, most robust, nicest interface, etc.) is usually the least XS code. That is, write as much as possible in Perl not in C.

This means write Perl code to get your data into a format that is easy for C to handle. In this case I'd just make a big flat list of values for both input and output.

void
_do_matrix( char *in, long rows, long cols, char *out )
CODE:
  long i, j, a;
  double *in_matrix= (double *)in;
  double *out_matrix= (double *)out;
  double value;

  for(  i= 0;  i < rows;  i++  ) {
    for(  j= i-1;  j >= 0;  j--  ) {
      value= 0;
      for(  a= 0;  a <= cols;  a++  ) {
        value += in_matri­x[i*cols+a]*in_matrix[j*cols+a];
      }
      value /= (cols-1);
      out_matrix[i*rows+j]= val­ue;
    }
  }
[download]

and then the Perl wrapper:

sub do_matrix(\@) {
    my( $avMatrix )= @_;
    my( $rows )= 0+@$avMatrix;
    my( $cols )= 0+@{$avMatrix->[0]};
    my $in= pack "d*", map @$_[0..$cols-1], @$avMatrix;
    my $out= pack "d".($rows*$rows);
    _do_matrix( $in, $rows, $cols, $out );
    my @out= unpack "d*", $out;
    my @ret;
    while(  @out  ) {
        push @ret, [ splice( @out, 0, $rows ) ];
    }
    return \@ret;
}
[download]

or something substantially similar after testing finds a few bugs.

                - tye