Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I posted this to comp.lang.perl.misc and have gotten nada thus far. Maybe some monkly advice can shine down on me...

What I need to do: send data over a network in network-byte order (big endian) in 4 byte chunks (XDR data format description, RFC 1832). Integers are 4 bytes long, strings of variable size are preceded by an int (4 bytes) which gives the real data length and then null padded to be a multiple of 4 bytes long (sorta pascal-esque), etc. A byte is defined as 8 bits.

Two problems (at least) I've found thinking through this:

  1. First I don't know how to use pack/unpack in a way that I can convert perl data into big-endian, binary forms of the correct sizes (I assume I need to use the "N" template for pack/unpack).
  2. I'd like to start working on string data as soon as I possibly can; I don't want to wait for the complete string to be in the buffer before I can pass it along to needy clients. Let's say I've encoded "Hello, world!" to an XDR string where the size is not known ahead of time. It should be 20 bytes long (first 4 bytes is an int containing the size of the string, 13, the next 16 bytes will be "Hello, world!\0\0\0"). Now say I have the first 8 bytes of this datagram, I decode the first 4 bytes and learn the string is 13 chars long, now what I would like to do is decode the next 4 bytes and get that much of the string ("Hell"). How can I pack the original perl string in a way that these 4 byte boundaries can be observed when unpacking?

This is perl-to-perl over the net for now but will need to be perl-to-C very soon.

Also wondering, can length, substr, join, // and other various perl string functions be used to look at/modify packed data/bytes safely? Any pitfalls here?

Lastly is there documentation on pack/unpack anywhere that describes in more detail than the Camel book (specificly I'd love to see a lot a lot of commented examples). pack/unpack has always been a time sink for me when I've needed to use it.

thanks,
Micah

  • Comment on How to use pack/unpack data over a network

Replies are listed 'Best First'.
Re: How to use pack/unpack data over a network
by repson (Chaplain) on Jan 05, 2001 at 16:18 UTC
    I'm no networking guru but here's some untested code that just might do the required input processing reliably. 
    # $sock is open, probably is a IO::Socket of some type

my $total_read = 0;
my $full_data = '';
my $total;

while ( 1 ) {
    my $read = 0;
    while ($read < 4) {
        $read += read($sock,$data,4-$read,$read)
        # not sure if last arg should have a +1 on it
    }
    $total_read += $read;
    
    if (! $total) {
        $total = unpack("N",$data);
    }
    elsif ( $total_read >= $total + 4 ) {
        process(substr($data,0,$total_read-4-$total));
        $full_data .= substr($data,0,$total_read-4-$total);
        last;
    }
    else {
        $full_data .= $data;
        process($data);
        other_process($full_data);
    }
}

full_process(substr($full_data,0,$total));

    [download]
    And the sender is 
    # $data has the required data and $sock is an open socket
print $sock pack('N',length($data)) . $data . ("\0" x (4-(length($data
+)%4)));

    [download]
[reply]
[d/l]
[select]
      On the sender's side we can go ahead and use the Z pack format, since we don't have to try processing it prematurely: 
      my $str_len = length($data);
$str_len += (4 - $str_len % 4 || 4);
print $sock pack("NZ$str_len", length($data), $data);

      [download]
      I started out thinking this would be a lot cleaner, but as you can see it doesn't look any better. *shrug*
[reply]
[d/l]
Re: How to use pack/unpack data over a network
by chipmunk (Parson) on Jan 05, 2001 at 20:02 UTC
    You can pack a null-padded ASCII string using the Z pack template: 
    $str = 'Hello, world!';
$len = length $str;

$packed_str = pack 'N', $len;

$len += 4 - (($len % 4) || 4);       # next multiple of 4

$packed_str .= pack "Z$len", $str;

# for testing:
($test_str = $packed_str) =~ s/([^\x20-\x7E])/sprintf '\\x%02x', ord $
+1/ge;
print "$test_str\n";

__END__

    [download]
    And the testing section prints the expected: \x00\x00\x00\x0dHello, world!\x00\x00\x00 Since you're packing the unpadded length in the first four bytes, you'll have to make it the next multiple of 4 before unpacking the string as well.
[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom