in reply to Optimise the script

Either remove some of your 16Gb of data, or write it in C. You also may consider caching the calls to UnixDate and Date_ConvTZ, but you then may risk the chance of a performance loss due to the additional memory (it depends on the number of different values).

I do notice two assignments to $pst_converted_date, but it's never used. And extracted_YMD is UnixDate(Date_ConvTZ(UnixDate(...))). That should be possible with a single UnixDate call. And the final match could be an eq.

Replies are listed 'Best First'.
Re^2: Optimise the script
by raybies (Chaplain) on Mar 30, 2011 at 13:40 UTC
    I'm with JavaFan. 16 GB is huge. That takes time. Maybe you could use multiple-cores/threads and break up the file into pieces, processing the pieces in parallel, or if you had a "cloud" architecture, farm out the pieces to a hundred different machines, but I'm inclined to say the problem is better rearchitected, before one goes into optimizing Perl.
[reply]

In Section Seekers of Perl Wisdom