Re: Reading binary file in perl having records of different length

In general, your approach is fine. Perl function "read" uses streams which in turn use data caching. So, it is fast to read even by 2 bytes. If your files are only few megabytes large, then you may read them completely into string but I don't think it will improve speed, because in my tests, stream was still obtaining data in the same chunks size, so number of system calls didn't change.

One comment on your use of "pack/unpack". Somehow you overuse it. For example, when searching for eye catcher just do "next unless $buffer eq '==';". When converting binary length just do "my $length = unpack('s', $buffer);".

One more thing. You don't check the return value of "read". Especially when you read "$length - 4" bytes. The file might be corrupted and you'll never get desired number of bytes. Plus, you say, your records may contain incorrect length, then what would be you strategy for recovery in this situation? Potentially the length may point to the middle of the next record.

Comment on Re: Reading binary file in perl having records of different length

Replies are listed 'Best First'.
Re^2: Reading binary file in perl having records of different length by jaypal (Beadle) on Jun 17, 2014 at 15:40 UTC
Thanks andal. All great suggestions. The binary length suggestion didn't seem to work in my case. I want to capture highest nibble first. For example, the binary length in hex is 0x03 0x50 so I need to read next 848 bytes. However if I do unpack('s', $buffer) it returns 20483 bytes (reading the binary data as 0x50 0x03).	[reply]
Re^3: Reading binary file in perl having records of different length by Anonymous Monk on Jun 17, 2014 at 16:13 UTC
See pack patterns "n" and "v" instead, or the "<" and ">" modifiers if you really want signed interpretation. `$ perl -le 'print unpack("v","\x03\x50")' 20483 $ perl -le 'print unpack("n","\x03\x50")' 848 $ perl -le 'print unpack("s<","\x03\x50")' 20483 $ perl -le 'print unpack("s>","\x03\x50")' 848` [download]	[reply] [d/l]
Re^4: Reading binary file in perl having records of different length by jaypal (Beadle) on Jun 17, 2014 at 16:33 UTC
Awesome!! The use of "n" pattern worked great.	[reply]


"be consistent"
	PerlMonks