in reply to Re: processing huge files
in thread processing huge files

If you're going to hand off the work to dd, you might want to use split, as it can act on full lines (so won't break in the middle of a record, given the logic the OP was using.).

You also don't need to recursively call it, as the equivalent to your dd example would be:

split -b 30720m -a 3 $INFILE

Replies are listed 'Best First'.
Re^3: processing huge files
by fauria (Deacon) on Aug 02, 2005 at 17:32 UTC
    I used dd because it can access directly a position in a file using skip, and then sequentialy read its content, without needing to load the whole file and then point to a location.
[reply]

In Section Seekers of Perl Wisdom