Test and endianness

andreas1234567 has asked for the wisdom of the Perl Monks concerning the following question:

(This relates to unpack and byteorder - tests fail on certain platforms).

The module Parse::Flash::Cookie comes with binary test files. These test files are collected from an Intel machine, and are thus little-endian, as described in perlport. These tests using these files obviously fail on big-endian platforms, such as sparc and powerpc.

$ uname -p
i686
$ perl -wle 'use Config; print $Config{byteorder}'
1234
[download]

Now I'm thinking that using these little-endian test files on a big-endian platform does not make sense, since these files would have been big-endian if they were originally saved on that platform. Is that correct?

For the module's next release, I have a number of options:

Make the module die on big-endian platforms. (Easy, but rude).
Skip testing on big-endian platforms (Tests more likely to pass, but there should be tests on all platforms).
Ship both little-endian and big-endian test files, and test depending on endianness (Not currently possible since I don't have access to any big-endian platforms, thus neither big-endian test files nor big-endian test environment).

Your thoughts please.

Update: Thu Jan 17 08:09:37 CET 2008: almut has confirmed Grandfather's assertion that SOL files are little-endian regardless of platform. almut also points out how to interpret floating point numbers on big-endian platforms. Thank you all very much for answering.

--
Andreas

Comment on Test and endianness Download Code

Replies are listed 'Best First'.
Re: Test and endianness by GrandFather (Saint) on Jan 16, 2008 at 11:00 UTC
It seems to me from a little browsing around the web that the LSO files are "network" files (little endien) and the endieness of the system storing the files is irrelevant (to the content of the file). The LSO parser should use little endien pack/unpack in its processing regardless of the orientation of the system the code is running on. That being the case, the tests should run against little endien LSO files (because that is the only sort that there is) and should report failures if the module is failing to convert from little endien to native endien for internal processing. Perl is environmentally friendly - it saves trees	[reply]
Re^2: Test and endianness by andreas1234567 (Vicar) on Jan 16, 2008 at 11:09 UTC
The LSO parser should use little endien pack/unpack This would work fine with `unpack("n")`, but that applies to integers only. Is this possibly also for floating point numbers without resorting to the Perl 5.9.2 `<` and `>` modifiers to unpack? I would prefer to be able to support Perl 5.8. -- Andreas	[reply] [d/l] [select]
Re^3: Test and endianness by almut (Canon) on Jan 16, 2008 at 14:45 UTC
As to the floats (I figure those are the values tagged as "number"): I just tried it on a big-endian machine (Solaris/sparc, flashplayer 9.0.47.0), in addition to little-endian Linux/x86. The results: Flashplayer apparently stores its cookies in the same format, independently of the architecture the player is running on¹ (which confirms what GrandFather wrote). I.e., a hexdump of the buffer (`unpack "H*", $buffer`) in your `_readFloat` routine consistently shows the same pattern, e.g. `4049000000000000` for 50.0 (as found in the `settings.sol` file, which is being created by default the first time the player is started). It seems you can properly decode those values using `unpack 'd', $buffer; # big-endian (e.g. sparc, PA-RISC) # i.e. 'perl -V' showing "byteorder=432 +1" unpack 'd', reverse $buffer; # little-endian (e.g. x86) # i.e. 'perl -V' showing "byteorder=123 +4"` [download] However, I so far haven't been able to test this with "more complex" floats, i.e. not just simple ints (like 50) represented as floats — mainly because I couldn't find a site which would set such a value... but I suppose it would work. (Do you have a link to a site which will make my flashplayer store numbers such as 726.127318273?) ___ ¹ Adobe++ for this, BTW — as it does allow you to share your home directory (~/.macromedia/) across multiple architectures. (Early versions of Firefox got this wrong, for example, storing architecture-dependent stuff in your profile, which made the browser segfault when you switched from one platform to another... (now fixed) )	[reply] [d/l] [select]
Re^4: Test and endianness by andreas1234567 (Vicar) on Jan 16, 2008 at 19:05 UTC
Re: Test and endianness by BrowserUk (Patriarch) on Jan 16, 2008 at 11:48 UTC
This is for the purposes of testing only. Correct? If so, then dodge the bullet. Store your test data file as ASCII. Within your test script, load the ASCII and pack it using the various "Native format" specifiers. This will generate your binary data correctly regardless of what platform you run it on. Supply the packed data to your test. Result: Your tests are platform independent, and you've exercised your binary interpretation routines. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply]
Re^2: Test and endianness by andreas1234567 (Vicar) on Jan 16, 2008 at 13:14 UTC
This is for the purposes of testing only. Correct? Yes. Within your test script, load the ASCII and pack it using the various "Native format" specifiers. That's a very elegant solution indeed. However, I assume GrandFather is correct in his assumption that the SOL files are little-endian regardless of platform. If one uses "Native format" specifiers to create the test files, one would get big-endian test files on big-endian platforms. The tests would then no longer reflect real use of the module on those platforms. -- Andreas	[reply]
Re: Test and endianness by ikegami (Patriarch) on Jan 16, 2008 at 10:41 UTC
Another possibility is to adjust the order of the bytes read based on the ordering of the bytes of the magic number in the header. Basically the same idea as the BOM in UTF-16 docs.	[reply]
Re: Test and endianness by apl (Monsignor) on Jan 16, 2008 at 10:48 UTC
You could post a request for someone who does have access to a big-endian machine to help you test your release.	[reply]
Re^2: Test and endianness by DrHyde (Prior) on Jan 16, 2008 at 11:51 UTC
Such as me. Tracking down my email address shouldn't be difficult, and I have a guest account available on a big-endian machine for people to test their code.	[reply]