in reply to Help Please: FCS Binary 2 ASCII conversion

This is a barely there implementation that seems to handle the example file as far as it goes--which is just the HEADER, TEXT & DATA sections--but it might serve you as a starting point:

#! perl -slw use strict; use Data::Dump qw[ pp ]; my %tmpls = ( '1,2,3,4' => { I => { 16 => 'v*', 32 => 'V*' }, F => { 32 => 'f<*' }, D => { 64 => 'd<*' }, }, '4,3,2,1' => { I => { 16 => 'n*', 32 => 'N*' }, F => { 32 => 'f>*' }, D => { 64 => 'd>*' }, }, ); open FCS, '<:raw', 'c:\downloaded\FCSExtract-TestData\FloatData4031.fc +s' or die $!; sysread( FCS, my $header, 58 ) or die $!; my( $id, $sText, $eText, $sData, $eData, $sAnalysis, $eAnalysis ) = unpack 'A6 x4 (A8)6', $header; sysseek FCS, $sText, 0 or die $!; sysread( FCS, my $text, $eText - $sText ) or die $!; my $delim = substr $text, 0, 1, ''; my %text = split $delim, $text; pp \%text; sysseek FCS, $sData, 0 or die $!; sysread( FCS, my $data, $eData - $sData ) or die $!; my $tmpl = $tmpls{ $text{ '$BYTEORD' } } { $text{ '$DATATYPE' } } { $text{ '$P1B' } }; die 'Can\'t handle format' unless $tmpl; my @data = unpack $tmpl, $data; pp \@data;

It produces output like this:

C:\test>FCS { "\$BEGINANALYSIS" => 0, "\$BEGINDATA" => 1794, "\$BEGINSTEXT" => 0, "\$BYTEORD" => "4,3,2,1", "\$DATATYPE" => "F", "\$ENDANALYSIS" => 0, "\$ENDDATA" => "441793 "\$ENDSTEXT" => 0, "\$FIL" => "4031.fcs", "\$INST" => " ", "\$MODE" => "L", "\$NEXTDATA" => 0, ... [ "0.400000005960464", "63555.6015625", "8488.80078125", "1326.24011230469", "35157.2421875", "174.960006713867", "76.6800003051758", "1597.51013183594", "4796.81005859375", "-13.2799997329712", "146.910003662109", "0.5", "60448.80078125", "8012.5205078125", "1193.40002441406", "31358.880859375", "167.400009155273", "31.3200016021729", "1451.99011230469", "5587.5400390625", "-4.15000009536743", "101.259994506836", "0.899999976158142", "31077.6015625", "7533.00048828125", "429.840026855469", "11815.2001953125", "47.5200004577637", "16.2000007629395", "1017.57006835938", "3776.0302734375", "13.2799997329712", "92.129997253418", 1, "40213.203125", ...

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"I'd rather go naked than blow up my ass"

Replies are listed 'Best First'.
Re^2: Help Please: FCS Binary 2 ASCII conversion
by david_lyon (Sexton) on Apr 13, 2010 at 01:44 UTC
    Thanks so much for your help. The code generates the correct results only at the begining half way through it doesnt return the right results....any ideas?? Also the code doesnt work with the second example....any ideas? Thanks once again!

      Patience is a virtue.

Re^2: Help Please: FCS Binary 2 ASCII conversion
by david_lyon (Sexton) on Apr 13, 2010 at 00:53 UTC
    Thanks so much for your help. there are some errors that I cant figure how to fix: 1) towards the end of the output eg below it doesnt generate the correct ascii/text, it does at the start of the output eg: ... "-198200359124992", "-72.1289825439453", "0.0589035041630268", 970023616, "-1642.0791015625", "1.95093229345274e-20", "2.48127919077995e-41", "2.5298206257041e-29", "2.3312967490238e+25", "-2.52964307903735e-29", "3.69734402349445e+19", "-2.80729058346108e-35", "126837727232", "-3.24638382888599e+26", "6248.572265625", "2916415963136", "126304272384", ] 2) It doesnt work with the second example Can't handle format at ./833927.pl line 43. thank you so much for your help.
      1. I made the classic out-by-one error here:
        sysread( FCS, my $data, $eData - $sData ) or die $!;

        The start- and end-of data offsets are inclusive. Therefore to read the whole section you need $eData - $sData +1.

      2. The delimiter for the TEXT section pairs in the second example is '|'.

        This is a regex meta-character, which means the split to construct the hash:

        my %text = split $delim, $text;

        Screws up. The solution is to quotemeta the delimiter read from the first character of the TEXT section just in case:

        my %text = split quotemeta( $delim ), $text;
      3. In the second example, $P1R is set to 4096, for $DATATYPE:I & $P1B:16.

        This means that only the first 12 bits of the 16-bit values should be considered. And that once the data has been unpacked, it must be masked to 12 bits to ensure that you don't get out-of-range values from picking up "stray bits" in the upper 4-bits.

        It would be a bloody-minded data supplier that would pack 12-bit data in to 16-bit words without ensuring the up 4-bits are zeroed, but according to the specification he isn't obliged to do so. And their example demonstrates this.

        I've added thise code to deal with that:

        if( $text{ '$DATATYPE' } eq 'I' and defined $text{ '$P1R' } ) { my $mask = $text{ '$P1R' } - 1; $_ &= $mask for @data; }
      4. I also noticed that there is a value in the TEXT section $PAR that indicates how many values there are per event, so I've used this to display that number of values per line:
        my $par = $text{ '$PAR' }; for( my $i = 0; $i < @data; $i += $par ) { print join ' ', @data[ $i .. $i + $par - 1 ]; }
      5. You asked by .msg about the %tmpls hash.
        my %tmpls = ( '1,2,3,4' => { I => { 16 => 'v*', 32 => 'V*' }, F => { 32 => 'f<*' }, D => { 64 => 'd<*' }, }, '4,3,2,1' => { I => { 16 => 'n*', 32 => 'N*' }, F => { 32 => 'f>*' }, D => { 64 => 'd>*' }, }, );

        Three TEXT section key/value pairs identify the format of the binary data in the DATA section.

        1. $BYTEORD Identifies the byte ordering of the data as big or little endian using the text tags '1,2,3,4' or '4,3,2,1'.

          Other byteordering are apparently possible!

        2. $DATATYPE: Identifies the type of the data: {I}nteger; {F}loat ; {D}ouble.

          There is also an 'A' datatype I haven't attempted to cater for. Though it doesn't look too hard to do.

        3. $P1B: Identifies the binary size in bits of the data. 16/32/64 bits.

          This seems to be mostly for the 'I' datatype. 64-bit integers is a possibility. The datastructure makes it easy to cater for that.

        All of this information is in the spec! Other formats that I haven't catered for in my example are possible.

        I use the %tmpls structure to map those three values to an appropriate unpack template:

        my $tmpl = $tmpls{ $text{ '$BYTEORD' } } { $text{ '$DATATYPE' } } { $text{ '$P1B' } };
      6. There is a lot more useful information in the TEXT section hash.

        You should work through the specs, and work out what each of those key/value pairs is, and how you should be using them.

      Anyway, here's a modified version that incorporates the corrections outlined above, that handles both supplied samples.

      But it is still only example code intended to give you a starting point from which to progress!

      #! perl -slw use strict; use Data::Dump qw[ pp ]; my %tmpls = ( '1,2,3,4' => { I => { 16 => 'v*', 32 => 'V*' }, F => { 32 => 'f<*' }, D => { 64 => 'd<*' }, }, '4,3,2,1' => { I => { 16 => 'n*', 32 => 'N*' }, F => { 32 => 'f>*' }, D => { 64 => 'd>*' }, }, ); open FCS, '<:raw', 'c:\downloaded\FCSExtract-TestData\FloatData4031.fcs' # 'c:\downloaded\FCSExtract-TestData\CC4_067_BM.fcs' or die $!; sysread( FCS, my $header, 58 ) or die $!; my( $id, $sText, $eText, $sData, $eData, $sAnalysis, $eAnalysis ) = unpack 'A6 x4 (A8)6', $header; sysseek FCS, $sText, 0 or die $!; sysread( FCS, my $text, $eText - $sText ) or die $!; my $delim = substr $text, 0, 1, ''; my %text = split quotemeta( $delim ), $text; pp \%text; <>; sysseek FCS, $sData, 0 or die $!; sysread( FCS, my $data, $eData - $sData +1 ) or die $!; my $tmpl = $tmpls{ $text{ '$BYTEORD' } }{ $text{ '$DATATYPE' } }{ $tex +t{ '$P1B' } }; die 'Can\'t handle format' unless $tmpl; my @data = unpack $tmpl, $data; if( $text{ '$DATATYPE' } eq 'I' and defined $text{ '$P1R' } ) { my $mask = $text{ '$P1R' } - 1; $_ &= $mask for @data; } my $par = $text{ '$PAR' }; for( my $i = 0; $i < @data; $i += $par ) { print join ' ', @data[ $i .. $i + $par - 1 ]; }

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        Thank you so much for your help I appreciate it very much. It now works for CC4_067_BM.fcs but still doesnt give the correct results for FloatData4031.fcs I have no idea how to fix that, apologizes for the inconvenience.