Chunked Perl Serialization

viral has asked for the wisdom of the Perl Monks concerning the following question:

I am trying to serialize perl objects but my transport layer only allows X bytes per message. Therefore I was wondering if there is a way to split (on the dispatch side) and join (on the consumer side).

Can I use Storable and just do a split as defined here. And a join on the consumer side?

Comment on Chunked Perl Serialization

Replies are listed 'Best First'.
Re: Chunked Perl Serialization by ikegami (Patriarch) on Nov 05, 2010 at 19:52 UTC
The receiver is going to need to know when it has received a whole message. The following does so by prefixing the data with the length of the data. Sender: `my $msg = pack('N/a', nfreeze($data)); while (length($msg)) { send_msg_block(substr($msg, 0, $max_msg_size, '')); }` [download] Receiver: `while (length($buf) < 4) { read_more_into_buf($buf); } my $data_len = unpack('N', substr($buf, 0, 4, '')); while (length($buf) < $data_len) { read_more_into_buf($buf); } $data = thaw(substr($buf, 0, $data_len, ''));` [download] This doesn't address the issue the anonymous monk raised about UDP. That's for your `send_msg_block` and `read_more_into_buf` to handle.	[reply] [d/l] [select]
Re: Chunked Perl Serialization by Anonymous Monk on Nov 05, 2010 at 19:38 UTC
Sounds like you're using UDP, in which case, its not as simple as split/join, you need to keep track of order ..... basically everything TCP does for you for free	[reply]