in reply to Formatting a large number of records

Perhaps this is a silly question, but I have to ask it.

Why do you need to speed up this process? Don't you have to get it right once only? After that (I presume) you would be done.

If you are wondering how to do your testing without going through the whole file, I would cut out a 1000 line chunk of the file (say) and use that for my testing. Once most everything looks good, run your script against the whole file.

As far as I know, some very smart people have worked on getting Perl's "Read in a line and process it" code to run absolutely as fast as possible -- so I don't think tinkering with the buffering is going to have a significant impact on the speed.

How long does it take to run the entire file? Can you get access to a faster machine, or one with more memory?

--t. alex
but my friends call me T.
  • Comment on Re: Formatting a large number of records

Replies are listed 'Best First'.
Re: Re: Formatting a large number of records
by elbow (Scribe) on Dec 30, 2002 at 15:44 UTC
    It's a valid question t. alex.

    Unfortunately this formatting has to be done between the end of one finance period and the start of the next and there is only a short period of time to get it run (a few hours).

    As for a better machine - not where I work! :-{

    elbow
[reply]

      Well, is Perl fast enough for the job? If not (your call, I guess), you'll have to code the equivalent functionality in C, which should give you a bit of an improvement in speed.

      What machine are you running this on? How long does the script take to run? I can't remember any benchmarks, but I know I've processed large files before and Perl whizzes through them. What you've discussed is not any heavy processing, so I can't imagine that your script would take more than a few minutes to process 1,000,000 lines.

      --t. alex
      but my friends call me T.
[reply]

In Section Seekers of Perl Wisdom