First I make pointer values like this

If SvLEN is non-zero, the PV belongs to Perl, and Perl will free it. This could be the cause of the segfault you had. You should set SvLEN to zero.

    SV* sv = newSV_type(SVt_PV);
    SvPV_set(sv, (char*)u_ptr);
    SvLEN_set(sv, 0);              /* So Perl won't free it. */
    SvCUR_set(sv, sizeof(Uint32));
    SvPOK_on(sv);
[download]

That way, if Perl reallocates the PV or if the scalar goes out of scope, it won't free the existing PV first. It's up to you to free surface->pixels.

AV * surfacex_pixel_array ( surface )

Perl functions can't return anything but a list of scalar. Specifically, you can't return an array. You need to return a reference to the array, or you need to flatten the array.

SV * construct_p_matrix ( SDL_Surface *surface )
{
    ...
    return newRV_noinc(matrix);
}
[download]

The problem is in setting the value of this pointer value.

use strict;
use warnings;

use Inline C => <<'__EOI__';

/* Something Perl shouldn't free */
static char s[4] = "\x11\x11\x11\x11";

SV* fetch() {
   SV* sv = newSV_type(SVt_PV);
   SvPV_set(sv, s);
   SvPOK_on(sv);
   SvLEN_set(sv, 0);
   SvCUR_set(sv, 4);
   return newRV_noinc(sv);
}

__EOI__

my $ref = fetch();
print(unpack('H*', $$ref), "\n");

$$ref = pack('N', 0x22222222);                 # Bad: Replaces the
$ref = fetch();                                #      PV instead of
print(unpack('H*', $$ref), "\n");              #      modifying it.

substr($$ref, 0, 4, pack('N', 0x33333333));    # OK
$ref = fetch();
print(unpack('H*', $$ref), "\n");

substr($$ref, 0, 4) = pack('N', 0x44444444);   # OK
$ref = fetch();
print(unpack('H*', $$ref), "\n");

substr($$ref, 0) = pack('N', 0x55555555);      # OK
$ref = fetch();
print(unpack('H*', $$ref), "\n");

vec($$ref, 0, 32) = 0x66666666;                # OK
$ref = fetch();
print(unpack('H*', $$ref), "\n");
[download]

11111111
11111111
33333333
44444444
55555555
66666666
[download]

I'm using a reference because my $sv = f(); would copy the string if I returned the scalar directly. You don't need to place refs in your array since the array acts as the reference. For example, the following should work if the array is setup properly:

vec($surf32_matrix->[0][0], 0, 32) = 0x12345678;
[download]

That said, your approach seems very fragile.