in reply to Re: How to split big files with Perl ?
in thread How to split big files with Perl ?

I'm sorry but this is really not good.

Aside from the fact that it doesn't compile, what is my $split_fh = "$fh" . 'split'; supposed to do? print $buf $outfile; or opening $out_file in read mode are pretty obvious errors. No error handling on open or read is also not great.

Do you think that reading 2GB of the input file into memory at a time is a very efficient way to go about it?

What happens when the size of the file is not exactly divisible by 5?

Replies are listed 'Best First'.
Re^3: How to split big files with Perl ?
by james28909 (Deacon) on Dec 26, 2014 at 19:01 UTC
    Well honestly it was like I said, it was purely untested code and was just for an example. I did not intend on it being a copy and paste example. All this does is reads the file then makes another file on the fly appending 001++ to it, thats all. I will however revise it and make any corrections so user can copy and paste it.
    I stand corrected, it will take more than what i posted to be able to split it up. What I was considering was takeing a 10gb file, and split it into exactly 4gb chunks. I think that would require to read 1 byte at a time, and write the buf to outfile until a counter reaches the 4gb limit. That way it is not filling the memory with all this data at one time and would work smoothly. Ill see what i can cook up.
[reply]

      So perhaps you should adopt a policy of testing before posting?

      /me thinks so.



      check Ln42!

[reply]
[d/l]
      This code has too many obvious issues to be left without comment.

      Actually too many to be even listed.

      That's not a bashing - it's a warning for everybody who tries to copy that.

      Cheers Rolf

      (addicted to the Perl Programming Language and ☆☆☆☆ :)

[reply]

      I see you've updated your original node, but haven't marked it as updated. Please don't do that since now some of the replies don't make sense anymore, see here.

      This is still not correct: my $split_fh = $fh . 'split'; #creates 'filename.split' - don't use $fh, the filehandle, use a variable containing the filename here. Also print $out_file, $buf; - remove the comma.

      Reading the file one byte at a time is a slightly better approach than 2GB at a time, but probably still not very efficient - a buffer size of at least a few kilobytes will probably get you better performance.

[reply]
[d/l]
[select]

      Pseudocode is fine to demonstrate a concept, unfortunately, your concept needs some improvements to be practical, as mentioned above.

      I will however revise it and make any corrections so user can copy and paste it.

      I think that is an excellent exercise!

[reply]
        Even revising it to "working state" produces errors as well. I wrote it very quickly as just an example anyway. Most of the code i post i do not plan on anybody using in their code, if anything it was just an example.
[reply]

In Section Seekers of Perl Wisdom