I wanted to prevent it "falling back" into character mode and *always* return byte offsets.

By using bytes, you do exactly the opposite.

require bytes;

$x = "\xC9\xCA\xCB\xCC";

utf8::downgrade($x);
print(substr($x,1,1)        eq "\xCA" ?1:0,"\n");  # 1
utf8::upgrade($x);
print(substr($x,1,1)        eq "\xCA" ?1:0,"\n");  # 1

utf8::downgrade($x);
print(bytes::substr($x,1,1) eq "\xCA" ?1:0,"\n");  # 1
utf8::upgrade($x);
print(bytes::substr($x,1,1) eq "\xCA" ?1:0,"\n");  # 0
[download]

bytes gives access to the internal storage format of the string. It has nothing to do with whether the string only contains bytes or not.

bytes::substr will probably do what you want. substr definitely will.

Further I thought binmode() is more a Win32 thing and as my code won't ever hit the MS world, I seldomly use it.

Prevents CRLF translations when the :crlf layer is used. Normally just on Windows.
Removes :encoding layers to prevent decoding. Shouldn't be there, but you're the one who's worried.

Just for completeness, my former code updated:

I don't know why you're asking for help (or why I'm giving it) if you sticking with that complex, buggy code when the simpler solution even does error checking.

Despite prompting, you never indicated whether it matters where you leave the file pointer when you're done. Does it?