I'm using a sub which expects as parameter a string describing a layer. [...] Is it sufficient to just use the empty string, ''

It doesn't matter what you pass to your sub. What matters is what you pass to open. Passing an empty string to open (for the second argument) is an error. You also need to specify the mode.

>perl -e"open my $fh, '', 'foo'"
Unknown open() mode '' at -e line 1.
[download]

Not specifying any layers causes open to use the default (hardcoded, from $ENV{PERLIO} or from use open).

sub myopen {
   my (..., $perlio, ...) = @_;
   $perlio ||= '';

   ...
   open(my $fh, ">$perlio", ...) or die ...;
   ...
}

myopen(..., ':raw:perlio', ...);  # bin mode with buffering
myopen(..., '',            ...);  # default
myopen(...,                   );  # default
[download]

You can set the default IO layer in $ENV{PERLIO} if you have a perl with perlio.

$ perl -wle'print"\x{20ac}"'
Wide character in print at -e line 1.
€
$ env PERLIO=:utf8 perl -wle'print"\x{20ac}"'
€
$
[download]

{ #default
  myfoo(...);
}

{ #override
  use open ':encoding(UTF-8)';
  myfoo(...);
}
[download]

Thanks for clarification. I fixed the function which actually calls the open, so that passing a layer is now optional.