Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi

i need to split a large file into smaller files

the large file contain 500 records (lines) i want to split it

every 30 lines in one file so it's going to be about 17 files

my problem is in the last there's no 30 lines if you know what i mean ..

Replies are listed 'Best First'.
Re: spliting large file into smaller
by Roy Johnson (Monsignor) on Nov 10, 2005 at 23:23 UTC
Re: spliting large file into smaller
by ikegami (Patriarch) on Nov 10, 2005 at 23:09 UTC
    open(my $in, '<', "filein.dat") or die("Unable to open filein.dat: $!\n"); my @buf; my $file_count = 0; while (<$in>) { push(@buf, $_); if (@buf == 30) { open(my $fh_out, '>', "file${file_count}.dat") or die("Unable to create file${file_count}.dat: $!\n"); print $fh_out ($_) foreach @buf; @buf = (); ++$file_count; } } if (@buf) { open(my $fh_out, '>', "file${file_count}.dat") or die("Unable to create file${file_count}.dat: $!\n"); print $fh_out ($_) foreach @buf; }

    Update: Changed
    print($fh_out)
    to
    print $fh_out ($_)

      Here's a simpler version:

      open(my $fh_in, '<', "filein.dat") or die("Unable to open filein.dat: $!\n"); my $fh_out; my $file_count = 0; while (<$fh_in>) { if ($. % 30 == 1) { open($fh_out, '>', "file${file_count}.dat") or die("Unable to create file${file_count}.dat: $!\n"); ++$file_count; } print $fh_out ($_); }

      The redundancy was eliminated. The buffer was eliminated.

      You might need a close before the open.

Re: spliting large file into smaller
by davidrw (Prior) on Nov 11, 2005 at 02:35 UTC
    As mentioned, there is a unix split command:
    user@host ~$] split -l 30 bigfile.txt
    Can also do this in a perl oneliner (warning -- this one is very inefficient):
    perl -ne 'open FILE, ">>", "smallfile".int($./30).".txt"; print FILE $ +_; close FILE }' bigfile.txt
    Take a look at the $. entry in perlvar.
Re: spliting large file into smaller
by blue_cowdawg (Monsignor) on Nov 11, 2005 at 03:50 UTC
        i need to split a large file into smaller files

    Based on your criteria here is yet another solution:

    #!/usr/bin/perl -w ######################################################### use strict; use Tie::File; my @fin=(); my @fout=(); tie @fin,"Tie::File","bigfile.txt" or die $!; my $fcount=1; for(my $ix=0;$ix<=$#fin;$ix+=30){ my $end = ( $ix+29 <=$#fin ? $ix+29 : $#fin ); my $outfilename = sprintf("litte_file_%0.2d.txt",$fcount); tie @fout,"Tie::File",$outfilename or die $!; for(my $iy=$ix;$iy<=$end;$iy++){ push @fout,$fin[$iy]; } untie @fout; $fcount++; }
    Some file sizes for you:
    [pberghol@cowdawg splitter]$ wc -l bigfile.txt 500 bigfile.txt [pberghol@cowdawg splitter]$ wc -l litte_file_* 30 litte_file_01.txt 30 litte_file_02.txt 30 litte_file_03.txt 30 litte_file_04.txt 30 litte_file_05.txt 30 litte_file_06.txt 30 litte_file_07.txt 30 litte_file_08.txt 30 litte_file_09.txt 30 litte_file_10.txt 30 litte_file_11.txt 30 litte_file_12.txt 30 litte_file_13.txt 30 litte_file_14.txt 30 litte_file_15.txt 30 litte_file_16.txt 20 litte_file_17.txt 500 total


    Peter L. Berghold -- Unix Professional
    Peter -at- Berghold -dot- Net; AOL IM redcowdawg Yahoo IM: blue_cowdawg
Re: spliting large file into smaller
by SamCG (Hermit) on Nov 14, 2005 at 22:44 UTC
    I happened to have almost exactly the same issue today. Here's how I handled it:
    #! perl use strict; my ($file) = @ARGV; my $counter; my $file_name = 1; open OUT_FILE, "+>$file_name.xls"; open IN_FILE, "<$file"; while (<IN_FILE>){ $counter++; print OUT_FILE $_; if($counter%20000 == 0){ $file_name++; open OUT_FILE, "+>$file_name.xls"; } }
    update: I happened to want to split my file into 20,000-line files, obviously. I got 22 of them.