in reply to Re: What's the most efficient way to write out many lines of data?
in thread What's the most efficient way to write out many lines of data?

Thanks for the interesting example... I had to try it as the times you quoted seemed very fast. But on my P4 with 512Mb RAM, and <who knows?> disc speed your program creates the file in around 3.5 secs!

However the "converted file" only contained "\n"s.

I'm no guru, but I worked out that (I think) you need a split before the join, or you won't have a list that join requires.

print OUTFILE join (",",  (split /(XXXXXXXXXX)/)), "\n";

worked, sort of, for me - I couldn't get your /(.{,10})/ pattern to work, although I think I understand what it's trying to match - any 10 chars exactly and reflect those in the stream as well as the other characters - which in this case aren't any. The resulting file simply had "\n"s as the split didn't seem to find a match.

This then took around 20 seconds...

$ perl file.pl
Creating file took 3.40489602088928 seconds
Splitting file using regexp took 19.5581229925156 seconds

[download]

This results in lines containing collections of 10 sets of the following ",XXXXXXXXXX," so that 2 commas appear between adjacent groups of X's and at the beginning and end of each line.

Replies are listed 'Best First'.
Re: Re: Re: What's the most efficient way to write out many lines of data?
by flounder99 (Friar) on Jul 10, 2003 at 11:27 UTC
    D'oh!

    You'd think I'd at least look at the output file! I changed the regex line to 

    print OUTFILE join (",",  /(.{1,10})/g), "\n"

    [download]
    and I got the results: 
    Creating file took 9.90625 seconds
Splitting file using regexp took 12 seconds

    [download]
    a lot slower but nowhere near 16 minutes.

    --

    flounder

[reply]
[d/l]
[select]
      Yup! That did it.

      However this is an idiom I'm not familiar with - I've only got the Camel's head here at work (Perl in a Nutshell) and the join entry doesn't expand on it. It's a great example of DWIM!

      With my use of the explicit split, as I said, the split file had lines like:-

      ,XXXXXXXXX,,XXXXXXXXXX,,XXXXXXXXXX,,...

      however, when I changed it to your use - even using my beginner's pattern of /(XXXXXXXXXX)/g - it DWIM'd!

      If you (or another brother) have time I'd appreciate an "expanded" Perl baby-talk explanation - i.e. put in all the bits that aren't necessary so we can see what's been left out.

      I (think I) understand the default $_ as being the input, but it's the use of a pattern to generate list context that's new to me.

      Thanks in advance

      hagen

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom