in reply to Re: Help Please: FCS Binary 2 ASCII conversion
in thread Help Please: FCS Binary 2 ASCII conversion

Thanks so much for your help. there are some errors that I cant figure how to fix: 1) towards the end of the output eg below it doesnt generate the correct ascii/text, it does at the start of the output eg: ... "-198200359124992", "-72.1289825439453", "0.0589035041630268", 970023616, "-1642.0791015625", "1.95093229345274e-20", "2.48127919077995e-41", "2.5298206257041e-29", "2.3312967490238e+25", "-2.52964307903735e-29", "3.69734402349445e+19", "-2.80729058346108e-35", "126837727232", "-3.24638382888599e+26", "6248.572265625", "2916415963136", "126304272384", ] 2) It doesnt work with the second example Can't handle format at ./833927.pl line 43. thank you so much for your help.
  • Comment on Re^2: Help Please: FCS Binary 2 ASCII conversion

Replies are listed 'Best First'.
Re^3: Help Please: FCS Binary 2 ASCII conversion
by BrowserUk (Patriarch) on Apr 13, 2010 at 02:34 UTC
    1. I made the classic out-by-one error here:
      sysread( FCS, my $data, $eData - $sData ) or die $!;

      The start- and end-of data offsets are inclusive. Therefore to read the whole section you need $eData - $sData +1.

    2. The delimiter for the TEXT section pairs in the second example is '|'.

      This is a regex meta-character, which means the split to construct the hash:

      my %text = split $delim, $text;

      Screws up. The solution is to quotemeta the delimiter read from the first character of the TEXT section just in case:

      my %text = split quotemeta( $delim ), $text;
    3. In the second example, $P1R is set to 4096, for $DATATYPE:I & $P1B:16.

      This means that only the first 12 bits of the 16-bit values should be considered. And that once the data has been unpacked, it must be masked to 12 bits to ensure that you don't get out-of-range values from picking up "stray bits" in the upper 4-bits.

      It would be a bloody-minded data supplier that would pack 12-bit data in to 16-bit words without ensuring the up 4-bits are zeroed, but according to the specification he isn't obliged to do so. And their example demonstrates this.

      I've added thise code to deal with that:

      if( $text{ '$DATATYPE' } eq 'I' and defined $text{ '$P1R' } ) { my $mask = $text{ '$P1R' } - 1; $_ &= $mask for @data; }
    4. I also noticed that there is a value in the TEXT section $PAR that indicates how many values there are per event, so I've used this to display that number of values per line:
      my $par = $text{ '$PAR' }; for( my $i = 0; $i < @data; $i += $par ) { print join ' ', @data[ $i .. $i + $par - 1 ]; }
    5. You asked by .msg about the %tmpls hash.
      my %tmpls = ( '1,2,3,4' => { I => { 16 => 'v*', 32 => 'V*' }, F => { 32 => 'f<*' }, D => { 64 => 'd<*' }, }, '4,3,2,1' => { I => { 16 => 'n*', 32 => 'N*' }, F => { 32 => 'f>*' }, D => { 64 => 'd>*' }, }, );

      Three TEXT section key/value pairs identify the format of the binary data in the DATA section.

      1. $BYTEORD Identifies the byte ordering of the data as big or little endian using the text tags '1,2,3,4' or '4,3,2,1'.

        Other byteordering are apparently possible!

      2. $DATATYPE: Identifies the type of the data: {I}nteger; {F}loat ; {D}ouble.

        There is also an 'A' datatype I haven't attempted to cater for. Though it doesn't look too hard to do.

      3. $P1B: Identifies the binary size in bits of the data. 16/32/64 bits.

        This seems to be mostly for the 'I' datatype. 64-bit integers is a possibility. The datastructure makes it easy to cater for that.

      All of this information is in the spec! Other formats that I haven't catered for in my example are possible.

      I use the %tmpls structure to map those three values to an appropriate unpack template:

      my $tmpl = $tmpls{ $text{ '$BYTEORD' } } { $text{ '$DATATYPE' } } { $text{ '$P1B' } };
    6. There is a lot more useful information in the TEXT section hash.

      You should work through the specs, and work out what each of those key/value pairs is, and how you should be using them.

    Anyway, here's a modified version that incorporates the corrections outlined above, that handles both supplied samples.

    But it is still only example code intended to give you a starting point from which to progress!

    #! perl -slw use strict; use Data::Dump qw[ pp ]; my %tmpls = ( '1,2,3,4' => { I => { 16 => 'v*', 32 => 'V*' }, F => { 32 => 'f<*' }, D => { 64 => 'd<*' }, }, '4,3,2,1' => { I => { 16 => 'n*', 32 => 'N*' }, F => { 32 => 'f>*' }, D => { 64 => 'd>*' }, }, ); open FCS, '<:raw', 'c:\downloaded\FCSExtract-TestData\FloatData4031.fcs' # 'c:\downloaded\FCSExtract-TestData\CC4_067_BM.fcs' or die $!; sysread( FCS, my $header, 58 ) or die $!; my( $id, $sText, $eText, $sData, $eData, $sAnalysis, $eAnalysis ) = unpack 'A6 x4 (A8)6', $header; sysseek FCS, $sText, 0 or die $!; sysread( FCS, my $text, $eText - $sText ) or die $!; my $delim = substr $text, 0, 1, ''; my %text = split quotemeta( $delim ), $text; pp \%text; <>; sysseek FCS, $sData, 0 or die $!; sysread( FCS, my $data, $eData - $sData +1 ) or die $!; my $tmpl = $tmpls{ $text{ '$BYTEORD' } }{ $text{ '$DATATYPE' } }{ $tex +t{ '$P1B' } }; die 'Can\'t handle format' unless $tmpl; my @data = unpack $tmpl, $data; if( $text{ '$DATATYPE' } eq 'I' and defined $text{ '$P1R' } ) { my $mask = $text{ '$P1R' } - 1; $_ &= $mask for @data; } my $par = $text{ '$PAR' }; for( my $i = 0; $i < @data; $i += $par ) { print join ' ', @data[ $i .. $i + $par - 1 ]; }

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      Thank you so much for your help I appreciate it very much. It now works for CC4_067_BM.fcs but still doesnt give the correct results for FloatData4031.fcs I have no idea how to fix that, apologizes for the inconvenience.
        but still doesnt give the correct results for FloatData4031.fcs

        Could you be a little more specific? How are the values wrong?

        Ie. Post a few (one or two) lines of the output + what they should be. And don't forget the <code></code> tags this time.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.