To elaborate a bit on the flag, it only tells perl’s string implementation whether it should be processing the string using ascii rules or using utf8 rules. If you change the flag you’re probably just breaking things unless you know for a fact that the ascii bytes of the string are in fact a valid utf8 sequence of bytes. If you want to take a string of latin1 characters and make sure they are represented as utf8 before handing those bytes to Python, you should be calling utf8::upgrade, and if you read some utf8 from a file handle and want the string to understand that it contains characters and not bytes, you should call utf8::decode. Because perl uses utf8 internally, calling utf8::decode doesn’t actually change any bytes (the first time you call it) and is just like calling utf8_on except that it also verifies that the string contains valid utf8.

Edit: Whoops, I didn’t read that carefully enough. I was expecting that the module you linked was recursively setting the utf8 flag in the manner of SvUTF8_on. Actually it does call upgrade, and the author just chose the name poorly.

It sounds sort of like you are saying that Inline::Python refuses to serialize string data unless it has the utf8 flag set on the string. This sounds like a bug in Inline::Python. The correct serialization pattern would be to upgrade the string as it was getting serialized, preferably only on the bytes being moved and without altering the original SV.

For your code example, it appears the only way to upgrade hash keys (in pure perl) is to rebuild the hash:

for (keys %h) { utf8::upgrade($_); $h{$_}= $h{$_} }
[download]

If you can change the original hash, I suggest this code snippet, which deletes entries with the non-utf8 keys and reuses the values for the utf8-version key.

for (keys %oldhash) {
    # change key and keep the value
    $oldhash{utf8_on($_)} = delete $oldhash{$_};
}
call_pyfunc(\%oldhash);
[download]

Neither _utf8_on (in Encode.pm) nor utf8_on (in DSU) mention a return value, _utf8_on (in DSU, probably different from Encode's) explicitly says

The data structure is converted in-place and as a convenience the passed variable is returned from the function.

Given mpersico's observation, it looks like for the hash itself it is not relevant whether its keys have an utf8 flag set or not.

Interesting point. That's not what I did though. My solution was

$ref->{ utf8_on($key) } = $ref->{$key};
[download]

and yet, I did not double the size of the hash, as proven by tests I have written to count and enumerate the keys as received in Python from Perl. I believe that my code "works" without duplicating the keys because utf8_on is not changing or manipulating the actual string that is the key; all it is doing is manipulating the metadata.

use Data::Structure::Util qw(get_refs utf8_on);

sub _perl_to_python_utf8_on {
    # We need to convert the hash keys ourselves.

    # 'get_refs' digs through the data structure
    # and returns an array ref of every reference
    # in the structure, at all levels.
    # All Hail CPAN; so glad I didn't have to walk
    # the data myself.
    my $refs = get_refs( $_[0] );

    # For each ref, process just the hashes.
    for my $ref ( @{$refs} ) {
        if ( ref($ref) eq 'HASH' ) {
            for my $key ( keys %$ref ) {
                # See text below.
                $ref->{ utf8_on($key) } = $ref->{$key};
            }
        }
    }

    # Let the utility convert everything else.
    return utf8_on( $_[0] );
}
[download]

$ref->{ utf8_on($key) } = delete $ref->{$key}.
[download]