How do I unpack Perl floats in C

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Howdy all, I have a medium sized application that I'm optimizing for speed (don't worry, it's actually necessary). Having profiled the code, I've found that a major hotspot is in a read loop where results are being read in and unpacked:

my @row = unpack("FFFFL", $tchandle->get($curdate));
[download]

The call to get is a major time sink (but not one i can get rid of). The call to unpack is another, lesser, but still significant time sink. I've already done all I can to minimize the number of reads. It occurs to me that since the format string never changes, a routine written in C that just unpacks the string into four floats and a long, without going through the internal overhead of the standard unpack call, might speed things up quite a bit. How would I go about writing a routine in C to unpack a string into four Perl floats and a long? It doesn't have to be especially portable (assume Intel hardware), and getting rid of the Perl floats and replacing them with normal floats is not an option.

Comment on How do I unpack Perl floats in C Download Code

Replies are listed 'Best First'.
Re: How do I unpack Perl floats in C by ikegami (Patriarch) on Apr 22, 2010 at 19:06 UTC
heh, `unpack` is actually faster than a custom XS solution (since it's not called via entersub???) `unpack: 1.200000 1.400000 1.800000 12345678.901230 2147483648 custom: 1.200000 1.400000 1.800000 12345678.901230 2147483648 nonportable: 1.200000 1.400000 1.800000 12345678.901230 2147483648 unsafe: 1.200000 1.400000 1.800000 12345678.901230 2147483648 Rate custom unsafe nonportable unpack custom 524225/s -- -6% -6% -20% unsafe 555156/s 6% -- -1% -16% nonportable 559449/s 7% 1% -- -15% unpack 658398/s 26% 19% 18% --` [download] (Some variation, but this is a typical run) You could save a itsy teeny bit by going to pure XS rather than using Inline::C, but I doubt it will be noticeable. use strict; use warnings; use Benchmark qw( cmpthese ); use Inline C => <<'__EOI__'; /* unpack("FFFFL", sv) / void custom_unpack(SV sv) { dXSARGS; NV f1; NV f2; NV f3; NV f4; U32 u; STRLEN len; char* buf = SvPVbyte(sv, len); if (len != sizeof(NV)4 + sizeof(U32)) croak("Length mismatch"); memcpy(&f1, buf, sizeof(NV)); buf += sizeof(NV); memcpy(&f2, buf, sizeof(NV)); buf += sizeof(NV); memcpy(&f3, buf, sizeof(NV)); buf += sizeof(NV); memcpy(&f4, buf, sizeof(NV)); buf += sizeof(NV); memcpy(&u, buf, sizeof(U32)); SP -= items; EXTEND(SP, 5); mPUSHn(f1); mPUSHn(f2); mPUSHn(f3); mPUSHn(f4); mPUSHu(u); XSRETURN(5); } / unpack("FFFFL", sv) / / Only works on machine where NV / / and U32 can be byte-aligned / void custom_unpack_nonportable(SV sv) { dXSARGS; STRLEN len; char* buf = SvPVbyte(sv, len); if (len != sizeof(NV)4 + sizeof(U32)) croak("Length mismatch"); SP -= items; EXTEND(SP, 5); mPUSHn((NV)buf); mPUSHn((NV)(buf + sizeof(NV))); mPUSHn((NV)(buf + sizeof(NV)2)); mPUSHn((NV)(buf + sizeof(NV)3)); mPUSHn((U32)(buf + sizeof(NV)4)); XSRETURN(5); } /* unpack("FFFFL", sv) / / Doesn't get magic / / Doesn't stringify / / Doesn't switch string format / / Doesn't do size checks / / Only works on machine where NV / / and U32 can be byte-aligned / void custom_unpack_unsafe(SV sv) { dXSARGS; char* buf = SvPVX(sv); SP -= items; EXTEND(SP, 5); mPUSHn((NV)buf); mPUSHn((NV)(buf + sizeof(NV))); mPUSHn((NV)(buf + sizeof(NV)2)); mPUSHn((NV)(buf + sizeof(NV)3)); mPUSHn((U32)(buf + sizeof(NV)4)); XSRETURN(5); } __EOI__ my %tests = ( 'unpack' => 'my @x = unpack("FFFFL", $buf);', 'custom' => 'my @x = custom_unpack($buf);', 'nonportable' => 'my @x = custom_unpack_nonportable($buf);', 'unsafe' => 'my @x = custom_unpack_unsafe($buf);', ); $_ = 'use strict; use warnings; our $buf; ' . $_ for values(%tests); local our $buf = pack('FFFFL', 1.2, 1.4, 1.8, 12345678.90123, 0x800000 +00); printf("unpack: %f %f %f %f %u\n", unpack("FFFFL", $buf)); printf("custom: %f %f %f %f %u\n", custom_unpack($buf)); printf("nonportable: %f %f %f %f %u\n", custom_unpack_nonportable($buf +)); printf("unsafe: %f %f %f %f %u\n", custom_unpack_unsafe($buf)); cmpthese(-2, \%tests); [download] Update*: Added `custom_unpack_unsafe`	[reply] [d/l] [select]
Re: How do I unpack Perl floats in C by almut (Canon) on Apr 22, 2010 at 18:15 UTC
Are you sure the `unpack` is the bottleneck? Running the following snippet on my pretty average hardware does around 800_000 unpacks per second. `my $buf = pack "FFFFL", (0.1) x 4, 999; for (1..1000_000) { my @row = unpack "FFFFL", $buf; } __END__ $ time ./836298.pl real 0m1.286s user 0m1.280s sys 0m0.004s` [download] I'd wager an XS routine doing the same wouldn't really be a lot faster...	[reply] [d/l] [select]
Re^2: How do I unpack Perl floats in C by Anonymous Monk on Apr 22, 2010 at 18:58 UTC
According to the profiler that I used, out of a 2 minuite execution time, bout 30 seconds is spent in calls to unpack.	[reply]
Re^3: How do I unpack Perl floats in C by ikegami (Patriarch) on Apr 22, 2010 at 19:30 UTC
So 20,000,000 million records? What do you do with the results of the unpack? You'll need to reimplement more than just a single opcode to get a benefit.	[reply]
Re: How do I unpack Perl floats in C by BrowserUk (Patriarch) on Apr 22, 2010 at 23:15 UTC
I've found that a major hotspot is in a read loop where results are being read in and unpacked: `my @row = unpack("FFFFL", $tchandle->get($curdate));` I bet if you wrote that as: `my $packed = $tchandle->get($curdate); my @row = unpack("FFFFL", $packed );` [download] and re-profiled, you'd find that it was the first line consuming the time, not the second. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. RIP an inspiration; A true Folk's Guy	[reply] [d/l] [select]