in reply to Re: Unpacking and converting
in thread Unpacking and converting

ikegami,

Not so. Simple addition results in warnings "Argument isn't numeric" whenever an empty string is there for a field that is nevertheless numeric.

Second option has no sense either, as I said that this data will be sent over the network. Considering that one line of data can easily be 2-4k (yes, kilo) and there can be like 5-7k lines per batch, savings from converting Unix time from 10-character strings to 4-byte integers are significant. And no, Storable does not convert the data by itself, I have checked before asking.

Anyway, the question was how to make array processing faster, not if there is any sense in doing it.

Regards,
Alex.

Replies are listed 'Best First'.
Re^3: Unpacking and converting
by ikegami (Patriarch) on Feb 15, 2011 at 21:05 UTC

    Simple addition results in warnings "Argument isn't numeric" whenever an empty string is there for a field that is nevertheless numeric.

    And your code warns for the string "abc" in a numeric field. You didn't perform validation, so neither did I.

    savings from converting Unix time from 10-character strings to 4-byte integers are significant.

    A measly 16% savings.

    $ perl -MDevel::Size=total_size -E'my @a; for (1..3000*6000/10) { my $ +item = "1234567890"; push @a, $item; } say(total_size(\@a))' 87588692 $ perl -MDevel::Size=total_size -E'my @a; for (1..3000*6000/10) { my $ +item = "1234567890"; $item = 0+$item; push @a, $item; } say(total_siz +e(\@a))' 73188692

    Anyway, the question was how to make array processing faster

    So why are you complaining about a little extra memory.

      You missed the point. I have no complaints about memory used, I have complaints about space taken by array frozen with Storable. Or rather, pushed to a socket with nstore(). My tests show ~50% reduction in size on actual data written to disk file against unprocessed text file, which is not so measly IMHO; considering that the data comes in bursts every three seconds or so, having its size reduced twice goes as significant in my book.

      Anyway, the question was: how do I make array processing go faster than does my code above?

      Regards,
      Alex.

        Wait, what? You are unpacking data just to turn around and re-pack it with nstore? It would go way faster if you did the unpacking on the other side of the socket. You wouldn't even have to use nstore.