I mistakenly thought that adding the :raw layer to the socket would set it up to accept any arbitrary bytes without worrying about what they were

You weren't mistaken. If you got that warning, it's that you didn't send bytes (≤255).

$ cat a.pl
use strict;
use warnings;
binmode(STDOUT, ':raw');
{ utf8::downgrade( my $x = chr(0xE9) ); print $x; }
{ utf8::upgrade(   my $x = chr(0xE9) ); print $x; }
{                  my $x = chr(0x100);  print $x; }

$ perl a.pl | od -t x1
Wide character in print at a.pl line 6.
0000000 e9 e9 c4 80
0000004
[download]

If you send text, you need to tell the socket how to convert the text into bytes. This type of serialisation is called character encoding. It can be done by calling encode or by using an :encoding layer.