Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

Is it possible at all to flush buffer and commit file write, if UTF-8 encoding is being used??

by dissident (Sexton)
on Oct 13, 2023 at 00:15 UTC ( [id://11154936]=perlquestion: print w/replies, xml ) Need Help??

dissident has asked for the wisdom of the Perl Monks concerning the following question:

I was trying unsuccessfully half the day to find out how to make Perl flush and commit write of a logfile, for showing the progress on a web page.

This is my code:

sub printtolog { my $txt = shift; my $logf; $txt = getdirdatetag() . ': ' . $txt . "\n"; # see https://perldoc.perl.org/functions/open open($logf, ">>:utf8", $logging_logfile) || die "$0: can't open $logging_logfile for appending: $!"; # see https://perldoc.perl.org/functions/select # this does not work # my $oldfh = select $logf; # $| = 1; # select($oldfh); # this also does not work # $logf->autoflush(1); print $logf $txt; # this does not work # $logf->flush; # $logf->sync; # this also does not work # $select()->flush(); # $select()->sync(); close $logf; }

Finally, I found a comment on StackOverflow, saying that the UTF-8 encoding layer adds another buffering layer.
(See the discussion between Ikegami and Ωmega on https://stackoverflow.com/questions/33812618/can-you-force-flush-output-in-perl)

Is this the reason why the logfile only gets created and written after either writing some kilobytes of many messages, or when the program much later terminates?

Does there exist a way to circumvent this buffering and to force creation and flushing of the file at all?

  • Comment on Is it possible at all to flush buffer and commit file write, if UTF-8 encoding is being used??
  • Download Code

Replies are listed 'Best First'.
Re: Is it possible at all to flush buffer and commit file write, if UTF-8 encoding is being used??
by kcott (Archbishop) on Oct 13, 2023 at 13:06 UTC

    G'day dissident,

    "This is my code:"

    It's some of your code. We're left guessing at the rest. For future reference, please supply an SSCCE.

    "I found a comment on StackOverflow, saying that the UTF-8 encoding layer adds another buffering layer."

    That refers to the "open pragma" (which you don't show) not the "open function" (which you do show).

    "Is this the reason why ..."

    That would depend on if, and how, you used the "open pragma" in the code you haven't shown us.

    A couple of other points regarding your post:

    "Does there exist a way to circumvent this buffering and to force creation and flushing of the file at all?"

    That emboldened text looks like you're shouting (at us). After "trying unsuccessfully half the day" I can understand that you're frustrated; please don't take it out on your fellow monks — we're here to help, not to be abused. Thankyou.

    With missing code, I am guessing somewhat; however, I think the following annotated script should be sufficient for you to resolve your problem.

    #!/usr/bin/env perl use strict; use warnings; use autodie; use utf8; use constant DELAY => 2; my @ascii_texts = map "text#$_", 0 .. 9; my @unicode_texts = map "&#127299;&#127284;&#127303;&#127299;#&#65039; +$_", qw{0&#65039; 1&#65039; 2&#65039; 3&#65039; 4&#65039; 5&#65039; 6 +&#65039; 7&#65039; 8&#65039; 9&#65039;}; for my $i (0 .. 9) { write_ascii_log($ascii_texts[$i]); write_unicode_log($unicode_texts[$i]); sleep DELAY; } close_ascii_log(); close_unicode_log(); sub now { return scalar localtime(); } # Log file routines # Note anonymous block { my ($asc_filename, $uni_filename); BEGIN { ($asc_filename, $uni_filename) = qw{ascii.log unicode.log}; } my ($ascii_fh, $unicode_fh); sub _open_asc { open $ascii_fh, '>>', $asc_filename; $ascii_fh->autoflush(); # For demo only: print "Run 'tail -f $asc_filename' in a separate window.\n"; print 'Hit <Enter> when ready: '; (undef) = scalar <STDIN>; return; } sub write_ascii_log { my ($text) = @_; _open_asc() unless defined $ascii_fh; $ascii_fh->print(now(), ": $text\n"); return; } sub close_ascii_log { return unless defined $ascii_fh; undef $ascii_fh; # For demo only: print "Stop 'tail -f $asc_filename'.\n"; print 'Hit <Enter> when ready: '; (undef) = scalar <STDIN>; return; } sub _open_uni { open $unicode_fh, '>>:encoding(UTF-8)', $uni_filename; $unicode_fh->autoflush(); # For demo only: print "Run 'tail -f $uni_filename' in a separate window.\n"; print 'Hit <Enter> when ready: '; (undef) = scalar <STDIN>; return; } sub write_unicode_log { my ($text) = @_; _open_uni() unless defined $unicode_fh; $unicode_fh->print(now(), ": $text\n"); return; } sub close_unicode_log { return unless defined $unicode_fh; undef $unicode_fh; # For demo only: print "Stop 'tail -f $uni_filename'.\n"; print 'Hit <Enter> when ready: '; (undef) = scalar <STDIN>; return; } }

    In case the line with Unicode text is rendered as entity references, here it is in a <pre> block:

    my @unicode_texts = map "🅃🄴🅇🅃#️$_", qw{0️ 1️ 2️ 3️ 4️ 5️ 6️ 7️ 8️ 9️};
    

    Here's a sample run:

    ken@titan ~/tmp/pm_11154936_buffering $ ./log_example1.pl Run 'tail -f ascii.log' in a separate window. Hit <Enter> when ready: Run 'tail -f unicode.log' in a separate window. Hit <Enter> when ready: Stop 'tail -f ascii.log'. Hit <Enter> when ready: Stop 'tail -f unicode.log'. Hit <Enter> when ready: ken@titan ~/tmp/pm_11154936_buffering $

    Here's the output from tailing ascii.log:

    ken@titan ~/tmp/pm_11154936_buffering $ tail -f ascii.log Fri Oct 13 23:07:47 2023: text#0 Fri Oct 13 23:08:07 2023: text#1 Fri Oct 13 23:08:09 2023: text#2 Fri Oct 13 23:08:11 2023: text#3 Fri Oct 13 23:08:13 2023: text#4 Fri Oct 13 23:08:15 2023: text#5 Fri Oct 13 23:08:17 2023: text#6 Fri Oct 13 23:08:19 2023: text#7 Fri Oct 13 23:08:21 2023: text#8 Fri Oct 13 23:08:23 2023: text#9

    Here's the output from tailing unicode.log:

    ken@titan ~/tmp/pm_11154936_buffering
    $ tail -f unicode.log
    Fri Oct 13 23:08:05 2023: 🅃🄴🅇🅃#️0️
    Fri Oct 13 23:08:07 2023: 🅃🄴🅇🅃#️1️
    Fri Oct 13 23:08:09 2023: 🅃🄴🅇🅃#️2️
    Fri Oct 13 23:08:11 2023: 🅃🄴🅇🅃#️3️
    Fri Oct 13 23:08:13 2023: 🅃🄴🅇🅃#️4️
    Fri Oct 13 23:08:15 2023: 🅃🄴🅇🅃#️5️
    Fri Oct 13 23:08:17 2023: 🅃🄴🅇🅃#️6️
    Fri Oct 13 23:08:19 2023: 🅃🄴🅇🅃#️7️
    Fri Oct 13 23:08:21 2023: 🅃🄴🅇🅃#️8️
    Fri Oct 13 23:08:23 2023: 🅃🄴🅇🅃#️9️
    

    Obviously, you'll need to run that yourself to see that the lines are printed to the logs every two seconds; you don't have to wait until filehandles are closed or the script ends.

    See also: "Suffering from Buffering?".

    — Ken

Re: Is it possible at all to flush buffer and commit file write, if UTF-8 encoding is being used??
by ikegami (Patriarch) on Oct 13, 2023 at 18:51 UTC

    my $oldfh = select $logf; $| = 1; select($oldfh); *does* flush Perl's buffer.

    $logf->autoflush(1); *does* flush Perl's buffer.

    $logf->flush(); *does* flush Perl's buffer.

    close $logf; *does* flush Perl's buffer.

    It doesn't necessary flush the OS's buffers. (You'll need to ask the OS to do that.) But the data will appear in the file to other processes.

Re: Is it possible at all to flush buffer and commit file write, if UTF-8 encoding is being used??
by swl (Parson) on Oct 13, 2023 at 03:32 UTC

    Is there a need to repeatedly open the log file? $logging_logfile seems to be a global variable, so why not instead have a filehandle in a global variable?

    That way you can open the file once, set autoflush to true, and then write as you go.

Re: Is it possible at all to flush buffer and commit file write, if UTF-8 encoding is being used??
by dissident (Sexton) on Oct 13, 2023 at 02:31 UTC

    Update:
    I have removed all the encoding stuff, and the problem remains.
    So the cause seems not the encoding, like I thought first.

    Maybe it has to do with the fact that the script is a script spawned by the CGI, via double-forking, setsid and exec?

    Now I ask myself, could this buffering issue be circumvented by not doing sequential file access, but instead random-access( eg. seek+print)?
    I will check that tomorrow and report back.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://11154936]
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others scrutinizing the Monastery: (4)
As of 2024-04-14 17:43 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found