in reply to Re^3: Unpacking and converting
in thread Unpacking and converting

You missed the point. I have no complaints about memory used, I have complaints about space taken by array frozen with Storable. Or rather, pushed to a socket with nstore(). My tests show ~50% reduction in size on actual data written to disk file against unprocessed text file, which is not so measly IMHO; considering that the data comes in bursts every three seconds or so, having its size reduced twice goes as significant in my book.

Anyway, the question was: how do I make array processing go faster than does my code above?

Regards,
Alex.

Replies are listed 'Best First'.
Re^5: Unpacking and converting
by ikegami (Patriarch) on Feb 16, 2011 at 17:07 UTC
    Wait, what? You are unpacking data just to turn around and re-pack it with nstore? It would go way faster if you did the unpacking on the other side of the socket. You wouldn't even have to use nstore.
[reply]
[d/l]
[select]

      My assumption is that the data comes initially from text; a file or other external source.

      Thus, despite that it is numeric data, when split to an array, each SV in the array is an SvPV with the numbers stored as ascii strings.

      But, if he runs over the array forcing the ascii to be stored in the IV or NV slot, then when Storable packs it, it only packs the IV/NV and not the PV, and so the Storeable image size is reduced.

      Of course, if the size of the transmission is really significant, then transmitting the original source text would probably be more efficient. Especially if he zipped it first.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
        Yeah. I understood that. That's why I made the same suggestion (minus zipping). Zipping is a good idea, though. It would slow down pre-processing, but it might speed things up overall by reducing the amount of packets to send.
[reply]

In Section Seekers of Perl Wisdom