Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re^6: Looking for alternative for IPC::Shareable (or increase size)

by jcb (Parson)
on Aug 07, 2020 at 22:31 UTC ( [id://11120485]=note: print w/replies, xml ) Need Help??


in reply to Re^5: Looking for alternative for IPC::Shareable (or increase size)
in thread Looking for alternative for IPC::Shareable (or increase size)

If I understand correctly, Sereal includes data compression, while Storable does not. Your test data is highly repetitive and therefore very compressible. Depending on deep details of your hardware, that compression may have made a significant difference by reducing the amount of data copied in some intermediate steps. (You did not output the length of $frozen, so I can only speculate here.) This may or may not be representative of our questioner's data or a meaningful comparison for non-trivial use.

There is some breakeven point below which the CPU overhead of attempting to compress the data will exceed the cost of simply copying the data. 1GiB of repeated base64 alphabet is obviously well above that point, but that point will vary with real-world data. Algorithms that perform better in the general large case usually have more overhead, and sometimes that can make a significant difference in smaller cases. As an example, I once did an analysis on some code I had written in C and found (to my surprise) that the actual data used was small enough that linear search would be faster than binary search — and that code was in an innermost loop where small gains are worthwhile.

  • Comment on Re^6: Looking for alternative for IPC::Shareable (or increase size)
  • Download Code

Replies are listed 'Best First'.
Re^7: Looking for alternative for IPC::Shareable (or increase size)
by marioroy (Prior) on Aug 10, 2020 at 05:20 UTC

    Hi, jcb

    I updated the examples to output the frozen length. MCE::Channel does not enable compression when using Sereal.

    > 1 billion chars (2^30 1073741824) Storable runtime : 2.106 seconds memory : 4,102 megabytes Sereal runtime : 1.549 seconds memory : 3,078 megabytes Sereal with compression enabled runtime : 2.170 seconds memory : 2,104 megabytes

    Thank you, DomX and jcb. ++ PerlMonks for this forum.

    Humble regards, Mario

      It seems obvious that the version using Storable is holding an extra copy of the data somewhere compared to the version using Sereal. The compression in Sereal is working reasonably well, reducing 1G of highly compressible input to about 52M stored.

      I still wonder if there really is a meaningful difference here — it should be equally possible to compress a frozen Storable image using any of the Compress:: modules to save memory or disk or network time, although it does appear that Sereal may have a slightly more efficient implementation (avoiding that extra copy of the data) that could be considered for adaptation to improve Storable as well. Maybe submit an enhancement request at the bug tracker for Storable?

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11120485]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others avoiding work at the Monastery: (4)
As of 2024-04-25 17:13 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found