Re^7: Looking for alternative for IPC::Shareable (or increase size)

I updated the examples to output the frozen length. MCE::Channel does not enable compression when using Sereal.

> 1 billion chars (2^30 1073741824)

Storable
    runtime : 2.106 seconds
    memory  : 4,102 megabytes

Sereal
    runtime : 1.549 seconds
    memory  : 3,078 megabytes
  
Sereal with compression enabled
    runtime : 2.170 seconds
    memory  : 2,104 megabytes
[download]

Thank you, DomX and jcb. ++ PerlMonks for this forum.

Humble regards, Mario

Comment on Re^7: Looking for alternative for IPC::Shareable (or increase size) Download Code

Replies are listed 'Best First'.
Re^8: Looking for alternative for IPC::Shareable (or increase size) by jcb (Parson) on Aug 11, 2020 at 02:08 UTC
It seems obvious that the version using `Storable` is holding an extra copy of the data somewhere compared to the version using `Sereal`. The compression in `Sereal` is working reasonably well, reducing 1G of highly compressible input to about 52M stored. I still wonder if there really is a meaningful difference here — it should be equally possible to compress a frozen `Storable` image using any of the `Compress::` modules to save memory or disk or network time, although it does appear that `Sereal` may have a slightly more efficient implementation (avoiding that extra copy of the data) that could be considered for adaptation to improve `Storable` as well. Maybe submit an enhancement request at the bug tracker for `Storable`?	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^8: Looking for alternative for IPC::Shareable (or increase size)
by jcb (Parson) on Aug 11, 2020 at 02:08 UTC

It seems obvious that the version using Storable is holding an extra copy of the data somewhere compared to the version using Sereal. The compression in Sereal is working reasonably well, reducing 1G of highly compressible input to about 52M stored.

I still wonder if there really is a meaningful difference here — it should be equally possible to compress a frozen Storable image using any of the Compress:: modules to save memory or disk or network time, although it does appear that Sereal may have a slightly more efficient implementation (avoiding that extra copy of the data) that could be considered for adaptation to improve Storable as well. Maybe submit an enhancement request at the bug tracker for Storable?

[reply]
[d/l]
[select]