xs outputs faster than perl can keep up

zentara has asked for the wisdom of the Perl Monks concerning the following question:

The node title suggests what I'm guessing the problem to be in this script. The xs portion of the script seems to output so fast, that newlines printed by Perl can't keep up, and the result is blocks of correct output, separated by blocks of newlines.

This is a script which divides a sequence of numbers by pi, using very high precision, provided by syphilis 's nice port of the Multiple Precision Float Library , which is at Math-MPFR-1.10.

I was making a comparison of speeds between c and Perl, and made 2 identical scripts, 1 c, and 1 Perl. The c script outputs as I expect, i.e 1 number per line. The Perl script outputs correct values, but the output comes in chunks of numbers, followed by chunks of newlines. If you look at the output loop, you can see that I print a newline after each number, but apperently Perl is saving them up and printing them all at once, instead of interleaving them into the output. This is totally bizarre behavior which I have never seen.

Can anyone explain why Perl isn't immediately putting the newline out, before continuing the loop?

#!/usr/bin/perl
use warnings;
use strict;
use Math::MPFR qw(:mpfr); 

use FileHandle;
$|++;

use constant MY_PI => 3.1415926535897932384626433832795028841971693993
+751;

Rmpfr_set_default_prec(256);
print Rmpfr_get_default_prec(),"\n"; 

my $n = Rmpfr_init2(256); # numerator                                 
+                        
my $d = Rmpfr_init2(256); # denominator                               
+                          
my $r = Rmpfr_init2(256); # result                                    
+                     

my @barray;

Rmpfr_set_d ($d, MY_PI , GMP_RNDD);                                   
+             
my $s3 = Rmpfr_get_str($d,10,0, GMP_RNDD); 
print "$s3\n";

for (my $i = 0; $i < 1000; $i++){

    $barray[$i] = Rmpfr_init2( 256 ); #init array element          

    Rmpfr_set_d ($n, $i , GMP_RNDD);                                  
+              

    Rmpfr_div ($barray[$i], $n, $d, GMP_RNDD); 
  }

 print "check mem, then hit a key\n"; 
 <>; 

for (my $i = 0; $i < 1000; $i++){
     Rmpfr_out_str($barray[$i],10,0,GMP_RNDD); 
     syswrite STDOUT ,"\n"; # print "\n" dosn't work either
     STDOUT->flush();
    }
    
__END__
[download]

The bad output looks something like this:

a bunch of numbers strung together
a bunch of blank lines
a bunch of numbers strung together
a bunch of blank lines
.......
......
[download]

I'm not really a human, but I play one on earth. Cogito ergo sum a bum

Comment on xs outputs faster than perl can keep up Select or Download Code

Replies are listed 'Best First'.
Re: xs outputs faster than perl can keep up by syphilis (Archbishop) on Feb 05, 2007 at 23:59 UTC
`Rmpfr_out_str($barray[$i],10,0,GMP_RNDD);` Does Rmpfr_get_str() work any better ? Or perhaps you can simply `print $barray[$i], "\n";` ('print' uses 'Rmpfr_get_str'). The overloaded functions use defaults, so you would want to first set the default rounding mode to GMP_RNDD if you use print(). I'll try to fix this over the next couple of days - I haven't yet managed to reproduce the problem on Win32 (and haven't yet tried on linux). Thanks for the report. Cheers, Rob Update: Included some code tags, missed earlier	[reply] [d/l] [select]
Re^2: xs outputs faster than perl can keep up by zentara (Cardinal) on Feb 06, 2007 at 13:11 UTC
Doh !! Yes, `print $barray[$i] ;` [download] does it properly. I don't know why I didn't try that, I guess I was thinking that it was an object, or I fell into the trap of trying to keep c syntax. Thanks. I'm not really a human, but I play one on earth. Cogito ergo sum a bum	[reply] [d/l]
Re^3: xs outputs faster than perl can keep up by syphilis (Archbishop) on Feb 06, 2007 at 13:59 UTC
However, I subsequently discovered that with the sample code you provided, 'Rmpfr_get_str' (and hence also 'print') output one less decimal place than 'Rmpfr_out_str'. Looks like there's a little bug there that needs fixing. Cheers, Rob	[reply]
Re: xs outputs faster than perl can keep up by almut (Canon) on Feb 05, 2007 at 22:54 UTC
As diotalevi said, I suspect this is another variant of "suffering from buffering", i.e. the XS code is using a different output buffer than the Perl code. I think some change will have to be made to the XS code: for example, (a) make it output the newlines itself (directly where it outputs the numbers), (b) make it use the same buffer that Perl's `print` or `syswrite` is using, (c) make sure the ouput from within XS is flushed every time after writing a number (this might still be tricky, though), or (d) change the API of Rmpfr_out_str() to return the number, so you can ouput everything synchronously from the Perl side... (I haven't looked at Math::MPFR - maybe there already is such a function...). (a) or (d) is probably easiest, with (a) being fastest...	[reply] [d/l] [select]
Re^2: xs outputs faster than perl can keep up by syphilis (Archbishop) on Feb 07, 2007 at 05:44 UTC
I think some change will have to be made to the XS code: for example, (a) make it output the newlines itself (directly where it outputs the numbers) That does fix the problem - ie in MPFR.xs replace `SV * Rmpfr_out_str(SV * p, SV * base, SV * dig, SV * round) { if(SvIV(base) < 2 \|\| SvIV(base) > 36) croak("2nd argument supplie +d to Rmpfr_out_str is out of allowable range (must be between 2 and 3 +6 inclusive)"); return newSVuv(mpfr_out_str(NULL, SvUV(base), SvUV(dig), (INT2PT +R(mpfr_t , SvIV(SvRV(p)))), SvUV(round))); }` [download] with `SV * Rmpfr_out_str(SV * p, SV * base, SV * dig, SV * round) { unsigned long ret; if(SvIV(base) < 2 \|\| SvIV(base) > 36) croak("2nd argument supplie +d to Rmpfr_out_str is out of allowable range (must be between 2 and 3 +6 inclusive)"); ret = mpfr_out_str(NULL, SvUV(base), SvUV(dig), (INT2PTR(mpfr_t +, SvIV(SvRV(p)))), SvUV(round)); printf("\n"); return newSVuv(ret); }` [download] On reflection, it probably doesn't make a lot of sense to be wrapping mpfr_out_str. I'll probably just leave Rmpfr_out_str as it is, and document the fact that calling it in a loop can lead to garbled output (and point out that it's safer to use Rmpfr_get_str). I mentioned elsewhere in this thread that Rmpfr_get_str currently can sometimes output one less digit than it ought (due to programmer error). The fix for that is to replace, in MPFR.xs (in the Rmpfr_deref2 function) `mpfr_get_str(out, expptr, b, n_dig, (INT2PTR(mpfr_t , SvIV(SvRV(p))) +), SvUV(round));` [download] with `mpfr_get_str(out, expptr, b, SvUV(n_digits), (INT2PTR(mpfr_t , SvIV( +SvRV(p)))), SvUV(round));` [download] That fix will find its way into the next version of Math::MPFR. Cheers, Rob	[reply] [d/l] [select]
Re: xs outputs faster than perl can keep up by diotalevi (Canon) on Feb 05, 2007 at 22:41 UTC
It sounds like one side is using a buffer and the other isn't. The perl-side syswrite is bypassing perl's normal buffer. Perhaps your XS isn't? ⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊	[reply]