in reply to Re: Re: Re: What's the most efficient way to write out many lines of data?
in thread What's the most efficient way to write out many lines of data?

Thanks a million for your input. I actually thought the regular expression to trim spaces might be a culprit, but I did not know that unpack would automatically trim trailing spaces. All the data I'm splitting is character data, so A should work just fine (if I understand unpack correctly.) :) I do have to ask, though, does unkpack trim leading spaces? That could be a problem, since some of the data in the fields is positional.

Put your speculation to rest...accumulating the records into a string and then writing in bulk does indeed backfire. It slows the process down...not much, but it does impact it negatively.

I was fairly certain Perl is optimized to buffer output efficiently, but I'm testing with 500K records and I need to really use this script on millions of records...so I want to get the time down as much as possible. :)

Thanks again for all the suggestions...I'll keep you posted as I implement your suggestions. :)

--Larry

  • Comment on Re: Re: Re: Re: What's the most efficient way to write out many lines of data?