Re^6: FCGI, tied handles and wide characters

Really glad to see you now have a working SSCCE. I'm somewhat bemused by the use of scalar ref for the string in the subroutine - can you elaborate on why that is desired or necessary?

Like most things Perl it's an embarrassing historical artifact, I cut'n'pasted from my usual file slurping routine which at a sprightly 50 lines was too big to fit in the SSCCE and didn't really think about it. Speaking of not thinking things through is there anything really wrong with the following SSCCE which accomplishes my original goal of bypassing perlio completely and using _utf8_on? Apart from the fact taint doesn't like it it seems like it would really speed up working on files that I know are utf-8.

use utf8;
use Encode qw(_utf8_on);
my $file = 'test_utf8';

binmode STDOUT, ':encoding(UTF-8)';
my $line = &sysfileread($file);
_utf8_on($line);

if ($line =~ /(❇)/) { print "found '$1'\n"; }
print $line;

sub sysfileread {
  my ($file) = @_;
  my $string;
  open(my $fh, "<", $file) || die "Can't open $file for reading: $!";
  my $size_left = -s $fh;
  while( $size_left > 0 ) {
    my $read_cnt = sysread($fh, $string, $size_left, length $string);
    last unless( $read_cnt );
    $size_left -= $read_cnt;
  }
  return $string;
}

Comment on Re^6: FCGI, tied handles and wide characters

Replies are listed 'Best First'.
Re^7: FCGI, tied handles and wide characters by hippo (Archbishop) on Sep 16, 2024 at 13:33 UTC
is there anything really wrong with the following It depends what you understand by "really wrong". It will run, but I would not choose to use it in production for these reasons: The `_utf8_on` subroutine comes with the caveat: The following API uses parts of Perl's internals in the current implementation. As such, they are efficient but may change in a future release. It would not be good if a future version suddenly broke it. The subroutine performs no validity checking on its input whatsoever. The first time it is fed non-utf8 input, it will corrupt your data (at best!). As stated, it won't run under taint mode. That should be some indication to you that it is not suitable for public use. Have you benchmarked it to see how much faster it really is compared with `Encode::decode()`? Always benchmark before optimising. 🦛	[reply] [d/l] [select]
Re^8: FCGI, tied handles and wide characters by Maelstrom (Beadle) on Sep 21, 2024 at 09:46 UTC
It might've been different with a bigger file but my benchmarking indicate considerably faster than Encode::decode() Rate implicit encode_decode utf_decode utf8_on implicit 34929/s -- -31% -60% -60% encode_decode 50765/s 45% -- -41% -43% utf_decode 86322/s 147% 70% -- -2% utf8_on 88314/s 153% 74% 2% -- It was a pleasant surprise to see `utf::decode` get so close though. Although given `utf::decode` won't protect me from non-utf8 input either I guess the optimal solution is `$line = Encode::decode('UTF-8', $line) unless (utf8::decode($line));`	[reply] [d/l] [select]
Re^9: FCGI, tied handles and wide characters by ikegami (Patriarch) on Sep 21, 2024 at 16:16 UTC
given utf::decode won't protect me from non-utf8 input either It does. It returns false.	[reply]
Re^7: FCGI, tied handles and wide characters by NERDVANA (Priest) on Sep 17, 2024 at 01:40 UTC
If you want maximum performance while still being safe, try utf8::decode. It logically decodes the string in-place, but because perl uses utf-8 internally the implementation just scans the string for utf-8 validity and then turns on the utf8 bit if it encountered any characters above 0x7F. You get the added benefit that it doesn't turn on the utf-8 bit if the entire string is 7-bit ascii, which can optimize code in various places when using that string. Likewise, utf8::encode is the fastest (safe) option for output. Just beware that they operate in-place, and may screw up assumptions made in the rest of the code if you didn't create a copy first, or discard the data afterward.	[reply]