in reply to Memory issue while opening and reading a file

You are trying to load in 1Gb of memory a whole 3Gb file. Don't be surprised if you can't.

The most immediate answer would be to avoid to load the file entirely (by using while (<TEST>) {---} instead of @records = <TEST>;). But even in that case, the resultant hash won't fit in memory.

Could you please tell us more about your <key>=<value>format? Are they numbers? In what range? Are strings? do they follow a defined format? Maybe another approach can be used with your data.

citromatik

Replies are listed 'Best First'.
Re^2: Memory issue while opening and reading a file
by koti688 (Sexton) on Apr 03, 2009 at 12:16 UTC
    keys and values or hexdata of any file. the file is created by this following script

    use Digest::SHA1 qw(sha1);
sub getHexData {
    my ($key)= unpack("H*",shift);
    return $key; 
}

print "Enter the file path :";
my $file = <STDIN>;
chomp($file);
print "Enter chunk size :";
my $size = <STDIN>;
chomp($size);

open(FILE, $file) || die "Unable to open the file";
open(TEST,">sample.txt") || die("This file will not open!");
$len = read(FILE, $data, $size);
while ( $len != 0) {
  $datalen = $size == $len ? $size: $len;
  $digest = sha1($data);
  $keydata = getHexData($digest);
  $valuedata = getHexData($data);
  print TEST "$keydata=$valuedata\n";
  $len = read(FILE, $data, $size);
}
close (TEST);
close(FILE);

    [download]
    which generates a flat textfile that consists of keys and values.Example of one pair is like below. The Value is of 8kb Data, when i give chunk size as 4000 , in my input.

    7070a40fcc1728fa940b683b169e4cee4d8204f0=255044462d312e330a25e2e3cfd30
+a342030206f626a0a3c3c0a2f54797065202f506167650a2f506172656e7420322030
+20520a2f436f6e74656e7473203130203020520a2f4d65646961426f78205b302e303
+0303020302e30303030203631322e30303030203739322e303030305d0a2f5472696d
+426f78205b302e3030303020302e30303030203631322e30303030203739322e30303
+0305d0a2f43726f70426f78205b302e3030303020302e30303030203631322e303030
+30203739322e303030305d0a2f5265736f7572636573203c3c0a2f50726f635365742
+05b2f504446202f496d616765435d0a2f584f626a656374203c3c202f496d31322031
+3220302052203e3e0a2f457874475374617465203c3c202f475331312031312030205
+2203e3e0a3e3e0a3e3e0a656e646f626a0a31302030206f626a0a3c3c0a2f46696c74
+6572205b2f466c6174654465636f6465205d0a2f4c656e677468203132370a3e3e0a7
+3747265616d0a78da33d4330001050334da10850fa3927379b90a79b9a03c734b2308
+03286a66688426988324680017324017c8e0e50ad7e2e5cae3e5d2770f363454482f0
+65b01d26a6a6c626c8ce1320b73433d3353532313055d033d23534b73055d234b3d0b
+73334bb0f3f43d730d8d145cf279b902c108612a955d0d7431005b383a94656e64737
+47265616d0a656e646f626a0a31322030206f626a0a3c3c0a2f54797065202f584f62
+6a6563740a2f53756274797065202f496d6167650a2f4e616d65202f496d31320a2f4
+6696c746572205b2f4443544465636f64655d0a2f576964746820313730320a2f4865
+6967687420323432320a2f42697473506572436f6d706f6e656e7420380a2f436f6c6
+f725370616365202f446576696365434d594b200a2f4c656e67746820363639363136
+0a3e3e0a73747265616d0affd8ffe000104a4649460001010100c800c80000ffee000
+e41646f626500640000000000ffdb004300130d0e110e0c13110f11151413171d301f
+1d1a1a1d3a2a2c2330453d4947443d43414c566d5d4c51685241435f82606871757b7
+c7b4a5c869085778f6d787b76ffc0001408097606a6044311004d11005911004b1100
+ffc4001f0000010501010101010100000000000000000102030405060708090a0bffc
+400b5100002010303020403050504040000017d010203000411051221314106135161
+07227114328191a1082342b1c11552d1f02433627282090a161718191a25262728292
+a3435363738393a434445464748494a535455565758595a636465666768696a737475
+767778797a838485868788898a92939495969798999aa2a3a4a5a6a7a8a9aab2b3b4b
+5b6b7b8b9bac2c3c4c5c6c7c8c9cad2d3d4d5d6d7d8d9dae1e2e3e4e5e6e7e8e9eaf1
+f2f3f4f5f6f7f8f9faffda000e0443004d0059004b00003f00e26b89ae26b89a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28

    [download]
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom