|
Re: Memory issue while opening and reading a file
by citromatik (Curate) on Apr 03, 2009 at 12:08 UTC
|
You are trying to load in 1Gb of memory a whole 3Gb file. Don't be surprised if you can't.
The most immediate answer would be to avoid to load the file entirely (by using while (<TEST>) {---} instead of @records = <TEST>;). But even in that case, the resultant hash won't fit in memory.
Could you please tell us more about your <key>=<value>format? Are they numbers? In what range? Are strings? do they follow a defined format? Maybe another approach can be used with your data.
| [reply]
[d/l]
[select] |
|
|
keys and values or hexdata of any file. the file is created by this following script
use Digest::SHA1 qw(sha1);
sub getHexData {
my ($key)= unpack("H*",shift);
return $key;
}
print "Enter the file path :";
my $file = <STDIN>;
chomp($file);
print "Enter chunk size :";
my $size = <STDIN>;
chomp($size);
open(FILE, $file) || die "Unable to open the file";
open(TEST,">sample.txt") || die("This file will not open!");
$len = read(FILE, $data, $size);
while ( $len != 0) {
$datalen = $size == $len ? $size: $len;
$digest = sha1($data);
$keydata = getHexData($digest);
$valuedata = getHexData($data);
print TEST "$keydata=$valuedata\n";
$len = read(FILE, $data, $size);
}
close (TEST);
close(FILE);
which generates a flat textfile that consists of keys and values.Example of one pair is like below. The Value is of 8kb Data, when i give chunk size as 4000 , in my input.
7070a40fcc1728fa940b683b169e4cee4d8204f0=255044462d312e330a25e2e3cfd30
+a342030206f626a0a3c3c0a2f54797065202f506167650a2f506172656e7420322030
+20520a2f436f6e74656e7473203130203020520a2f4d65646961426f78205b302e303
+0303020302e30303030203631322e30303030203739322e303030305d0a2f5472696d
+426f78205b302e3030303020302e30303030203631322e30303030203739322e30303
+0305d0a2f43726f70426f78205b302e3030303020302e30303030203631322e303030
+30203739322e303030305d0a2f5265736f7572636573203c3c0a2f50726f635365742
+05b2f504446202f496d616765435d0a2f584f626a656374203c3c202f496d31322031
+3220302052203e3e0a2f457874475374617465203c3c202f475331312031312030205
+2203e3e0a3e3e0a3e3e0a656e646f626a0a31302030206f626a0a3c3c0a2f46696c74
+6572205b2f466c6174654465636f6465205d0a2f4c656e677468203132370a3e3e0a7
+3747265616d0a78da33d4330001050334da10850fa3927379b90a79b9a03c734b2308
+03286a66688426988324680017324017c8e0e50ad7e2e5cae3e5d2770f363454482f0
+65b01d26a6a6c626c8ce1320b73433d3353532313055d033d23534b73055d234b3d0b
+73334bb0f3f43d730d8d145cf279b902c108612a955d0d7431005b383a94656e64737
+47265616d0a656e646f626a0a31322030206f626a0a3c3c0a2f54797065202f584f62
+6a6563740a2f53756274797065202f496d6167650a2f4e616d65202f496d31320a2f4
+6696c746572205b2f4443544465636f64655d0a2f576964746820313730320a2f4865
+6967687420323432320a2f42697473506572436f6d706f6e656e7420380a2f436f6c6
+f725370616365202f446576696365434d594b200a2f4c656e67746820363639363136
+0a3e3e0a73747265616d0affd8ffe000104a4649460001010100c800c80000ffee000
+e41646f626500640000000000ffdb004300130d0e110e0c13110f11151413171d301f
+1d1a1a1d3a2a2c2330453d4947443d43414c566d5d4c51685241435f82606871757b7
+c7b4a5c869085778f6d787b76ffc0001408097606a6044311004d11005911004b1100
+ffc4001f0000010501010101010100000000000000000102030405060708090a0bffc
+400b5100002010303020403050504040000017d010203000411051221314106135161
+07227114328191a1082342b1c11552d1f02433627282090a161718191a25262728292
+a3435363738393a434445464748494a535455565758595a636465666768696a737475
+767778797a838485868788898a92939495969798999aa2a3a4a5a6a7a8a9aab2b3b4b
+5b6b7b8b9bac2c3c4c5c6c7c8c9cad2d3d4d5d6d7d8d9dae1e2e3e4e5e6e7e8e9eaf1
+f2f3f4f5f6f7f8f9faffda000e0443004d0059004b00003f00e26b89ae26b89a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28a28
+a28a28a28a28a28a28a28a28a28a28a28a28
| [reply]
[d/l]
[select] |
Re: Memory issue while opening and reading a file
by cdarke (Prior) on Apr 03, 2009 at 12:33 UTC
|
Nothing to do with RAM, well, assuming you are not using something like 16-bit DOS anyway.
Operating Systems like Windows NT and onwards, UNIX, Linux, use virtual memory. On a 32-bit OS the largest address is 2**32, which gives a process address space of 4Gb. That sounds OK!
No, kernel reserves a large slice for itself. On 32-bit Windows that is 2Gb (unless you use a thing called Address Windowing Extensions).
Now, Perl uses the heap, which (on Windows) is again restricted, and a maximum is set at compile time. The default size on Microsoft Visual Studio (last time I looked) is only 1Mb. Details on Linux/UNIX differ, but the jist is the same.
If you were using 64-bit you might be able to get away with it, but somehow I doubt it.
| [reply] |
|
|
Don't forget that one byte in the file does not equate to one byte in memory here. I have seen figures quoted of a factor of 5-10 times overhead for Perl data.
| [reply] |
Re: Memory issue while opening and reading a file
by derby (Abbot) on Apr 03, 2009 at 16:41 UTC
|
As others have said, jamming 3G of data into 1G of ram is going to be problematic ... especially since
you're doubling your amount of memory by reading the file into memory first. I would recommend:
- processing the file line by line
open( my $fh, "<", $sigDataFile ) || die( "cannot open file - $!" );
while( <$fh> ) {
chomp;
my( $key, $value ) = split(/=/, $_); # does = need to be escaped?
...
}
- use something like SDBM_File or GDBM_File
use Fcntl;
use SDBM_File;
tie( %hash, 'SDBM_File', 'temp_file_name', O_RDWR|O_CREAT, 0666 )
or die "Couldn't tie SDBM file 'filename': $!; aborting";
open( my $fh, "<", $sigDataFile ) || die( "cannot open file - $!" );
while( <$fh> ) {
chomp;
my( $key, $value ) = split( /=/, $_ );
$hash{$key} = $value;
...
}
| [reply]
[d/l]
[select] |
Re: Memory issue while opening and reading a file
by Anonymous Monk on Apr 03, 2009 at 12:07 UTC
|
Looks like the line:
@records = <TEST>;
attempt to read all of the 3GB file into the perl process memory, where it will not fit.
Consider splitting your 3 GB file into smaller pieces. | [reply] |
Re: Memory issue while opening and reading a file
by shmem (Chancellor) on Apr 03, 2009 at 12:09 UTC
|
I want to pour 3 gallons of water into a 1 gallon bucket, and the water keeps flowing over! Why?
| [reply] |
Re: Memory issue while opening and reading a file
by vinoth.ree (Monsignor) on Apr 03, 2009 at 12:10 UTC
|
There is no fault in script.
Here it is problem
@records = <TEST>;
It is problem with memory, because your RAM size is less than your file size.
| [reply]
[d/l] |
Re: Memory issue while opening and reading a file
by rovf (Priest) on Apr 03, 2009 at 12:10 UTC
|
My $sigdata file is a flat text file about 3 Gb
...
My System details are like
Pentium(R) 4 CPU 2.26 GHz
1.00GB of RAM.
Swapping space nonwithstanding, isn't it a bit ambituous to try to squeeze a 3GB file into 1GB of RAM?
--
Ronald Fischer <ynnor@mm.st>
| [reply]
[d/l] |
