You could take the IP, XOR each element in succession to get a random seed, then randomly select three elements for RGB, then randomly XOR or AND the the fourth element with each of the selected RGB elements. This should have enough randomness to prevent RE of the IP and would (hopefully) produce something other than a nice even shade of grey.

use strict;
use warnings;

my @ips = qw(
    127.0.0.1
    192.172.24.23
    172.25.1.86
    66.35.250.150
    209.197.123.153
    66.39.54.27
    204.74.99.100
);

for my $ip (@ips) {
    my @ip = split /\./, $ip;
    my $seed = 0;
    $seed ^= $_ for @ip;
    srand($seed);
    my ($r,$g,$b,$mask) = map { splice @ip, int(rand(scalar @ip)), 1 }
+ (1 .. @ip);
    ($r,$g,$b) = map { int(rand(2)) ? $_ ^ $mask : $_ & $mask } ($r,$g
+,$b);

    printf("%-15s => %3d %3d %3d\n",$ip,$r,$g,$b);
}
[download]

This will produce the following:

127.0.0.1       =>   0   0   0
192.172.24.23   =>   4  16 215
172.25.1.86     => 250 173 181
66.35.250.150   =>   2 212  66
209.197.123.153 => 190 170 226
66.39.54.27     =>  45  38   2
204.74.99.100   =>  96  46 168
[download]

Here we have 2 coming from 66 in the Red column and 27 in the Blue column. I don't thing this is reverse-engineerable. And since we are using the IP to derive the random seed, we are guaranteed that the same IP will produce the same RGB value.