in reply to Re^2: Unpacking and converting
in thread Unpacking and converting

Simple addition results in warnings "Argument isn't numeric" whenever an empty string is there for a field that is nevertheless numeric.

And your code warns for the string "abc" in a numeric field. You didn't perform validation, so neither did I.

savings from converting Unix time from 10-character strings to 4-byte integers are significant.

A measly 16% savings. 

$ perl -MDevel::Size=total_size -E'my @a; for (1..3000*6000/10) { my $
+item = "1234567890"; push @a, $item; } say(total_size(\@a))'
87588692

$ perl -MDevel::Size=total_size -E'my @a; for (1..3000*6000/10) { my $
+item = "1234567890"; $item = 0+$item; push @a, $item; } say(total_siz
+e(\@a))'
73188692

[download]

Anyway, the question was how to make array processing faster

So why are you complaining about a little extra memory.

Replies are listed 'Best First'.
Re^4: Unpacking and converting
by dwalin (Monk) on Feb 15, 2011 at 21:31 UTC

    You missed the point. I have no complaints about memory used, I have complaints about space taken by array frozen with Storable. Or rather, pushed to a socket with nstore(). My tests show ~50% reduction in size on actual data written to disk file against unprocessed text file, which is not so measly IMHO; considering that the data comes in bursts every three seconds or so, having its size reduced twice goes as significant in my book.

    Anyway, the question was: how do I make array processing go faster than does my code above?

    Regards,
    Alex.

[reply]
      Wait, what? You are unpacking data just to turn around and re-pack it with nstore? It would go way faster if you did the unpacking on the other side of the socket. You wouldn't even have to use nstore.
[reply]
[d/l]
[select]

        My assumption is that the data comes initially from text; a file or other external source.

        Thus, despite that it is numeric data, when split to an array, each SV in the array is an SvPV with the numbers stored as ascii strings.

        But, if he runs over the array forcing the ascii to be stored in the IV or NV slot, then when Storable packs it, it only packs the IV/NV and not the PV, and so the Storeable image size is reduced.

        Of course, if the size of the transmission is really significant, then transmitting the original source text would probably be more efficient. Especially if he zipped it first.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom