stevehicks has asked for the wisdom of the Perl Monks concerning the following question:

Hi Guys, I'm hoping you can help because I've exhausted my understanding of this !! I'm writing a decoder for messages that I receive on a socket. The Messages look like this : 
B2t^?\251 cEy]\213\247\3350002900262O0 NL000634286722^F^Q^S\210^F^S^S\
+210

[download]
The rough guidance I have is that the format is :

Field Type Len

where "Alpha" is a string and Num is a binary field of type "uintx_t" (I guess uint8_t for example) I'm trying to unpack this and I can't seem to decode the MsgSeqNum and ClOrdID fields properly. I know the value of the messageType, ProtoOrVer and MsgLen fields because I can decode them successfully(verified with another program I have no source code access too)...however whatever I do I can't decode see to decode the MsgSeqNum and ClOrdID fields. I know from the output of the other program that the ClOrdID in this message is "4295075805" but no unpack strings I specify seem to work. If I use the following unpack statement just to get the field boundaries right : 
($messageType,$ProtoOrVer,$MsgLen,$MsgSeqNum,$ClOrdID,$OnBehalfOfCompI
+D,$rest) =unpack "A A n B32 B64 A*",$sourceMsg};

[download]
then I get : 
08:02:43 : messageType == B
08:02:43 : ProtoOrVer == 2
08:02:43 : MsgLen == 116
08:02:43 : MsgSeqNum == 00000000000000001111111010010101
08:02:43 : ClOrdID == 000001001100011010100010100111101011101011010001
+1110010110111011
08:02:43 : OnBehalfOfCompID == 00029002
08:02:43 : rest == 62O0 NL000634286722^F^Q^S\210^F^S^S\210

[download]
(This looks right because OnBehalfOfCompID is in the right place) Now, given that using "n" works for the MsgLen field(length of 2 from spec and n == 16 bits on pack perldoc page) I tried using n2 on the basis that the field is supposed to be twice as long but I get the following, which is obviously wrong : 
07:43:55 : messageType == B
07:43:55 : ProtoOrVer == 2
07:43:55 : MsgLen == 116
07:43:55 : MsgSeqNum == 0
07:43:55 : ClOrdID == 32681
07:43:55 : OnBehalfOfCompID == 8291
07:43:55 : rest == 17785

[download]
On the basis that the field has a length of 4 x 8 == 32 I have also tried using combinations of "S", "L" and "N", for example : 
($messageType,$ProtoOrVer,$MsgLen,$MsgSeqNum,$ClOrdID,$OnBehalfOfCompI
+D,$rest) =unpack "A A n N N2 A*",$sourceMsg};

[download]
to which I get : 
08:11:45 : messageType == B
08:11:45 : ProtoOrVer == 2
08:11:45 : MsgLen == 116
08:11:45 : MsgSeqNum == 32681
08:11:45 : ClOrdID == 543376761
08:11:45 : OnBehalfOfCompID == 1569433565
08:11:45 : rest == 00029002

[download]
..which is also clearly wrong. I've had 4 hours hours sleep, I'm at my wit's end and I'm up against the wall to get this nailed today. Any help GREATLY appreciated !!

Replies are listed 'Best First'.
Re: Lost in unpack hell
by ikegami (Patriarch) on Feb 11, 2010 at 09:29 UTC

    I tried using n2 on the basis that the field is supposed to be twice as long

    'N' is a unsigned 32-bit (4 byte) value using the same byte ordering as 'n'

    'Q>' is a unsigned 64-bit (8 byte) value using the same byte ordering as 'n'

    From the info you gave us, you want 

    unpack 'a a n N Q> a*'

    [download]
    which can also be written as 
    unpack 'a a S> L> Q> a*'

    [download]

    That said,

    I know from the output of the other program that the ClOrdID in this message is "4295075805"

    The long string you gave for ClOrdID translates to 04C6A29EBAD1E5BB(base16).

    You're expecting 4295075805(10) = 000000010001A7DD(16)

    No matter how you rearrange the hex digit pairs, those will never match. It's clearly not a byte ordering problem.

[reply]
[d/l]
[select]
      </cite>The long string you gave for ClOrdID translates to 04C6A29EBAD1E5BB(base16). You're expecting 4295075805(10) = 000000010001A7DD(16) No matter how you rearrange the hex digit pairs, those will never match. It's clearly not a byte ordering problem. </cite> Then that means that the other program is "lying" to me, which _is_ possible I guess since it appears to have other problems. I tried the unpack string you mentioned : 
      >> unpack 'a a n N Q> a*'

      [download]
      but I get : Invalid type 'Q' in unpack at /opt/devel/mdscripts/PerfAgent/lib/enxtUtpDirect.pm This is (I believe)because I'm not using a 64bit perl and I can't because of some PERL packages I am using. Given this, can I use something other than 'Q' ?? Thanks very much for the assistance, it is much appreciated
[reply]
[d/l]
[reply]

        Then that means that the other program is "lying"

        No, that does not necessarilly follow from what I said.

[reply]
Re: Lost in unpack hell
by Anonymous Monk on Feb 11, 2010 at 08:31 UTC
    If you could please include runnable code, something reusable, that shows original input, expected output, actual output, ex 
    my $input = pack 'H*', "your binary input unpacked with 'H*'";
my ($messageType,$ProtoOrVer,$MsgLen,$MsgSeqNum,$ClOrdID,$OnBehalfOfCo
+mpID,$rest) =unpack "A A n N N2 A*",$sourceMsg};

print ...
print "expected output\n";

    [download]
    thanks
[reply]
[d/l]

Back to Seekers of Perl Wisdom