in reply to Small Corrupted Storable Retrieve Dies With an Invalid "Out of memory!"

I have identified the errors in the file. There are 8 bytes that appear wrong. 7 added bytes (duplicate) and one missing byte by my calculation.

[root@devel3 Storable.patch]# cat /usr/bin/bdiff.pl #!/usr/bin/perl use Algorithm::Diff qw(traverse_sequences); my ( $f1, $f2 ) = @ARGV; open $fh1, $f1 or die $!; open $fh2, $f2 or die $!; local $/; my $seq1 = [split //, <$fh1>]; my $seq2 = [split //, <$fh2>]; my (@match,@d1, @d2) = ((),(),()); traverse_sequences( $seq1, $seq2, { MATCH => sub { push @match, $seq1->[$_[0]] }, DISCARD_A => sub { push @d1, printable2($_[0])." : - ".printable($ +seq1->[$_[0]]) }, DISCARD_B => sub { push @d2, printable2($_[1])." : + ".printable($ +seq2->[$_[1]]) }, }); sub printable { sprintf "%s 0x%02X", ord($_[0])>32?$_[0]:'.',ord $_[0] + } sub printable2 { sprintf "%08d 0x%08X", $_[0],$_[0] } print join "\n", @d1,@d2,"\n"; [root@devel3 Storable.patch]# bdiff.pl store.original store.patch 00004096 0x00001000 : - s 0x73 00004097 0x00001001 : - s 0x73 00004098 0x00001002 : - i 0x69 00004099 0x00001003 : - o 0x6F 00004100 0x00001004 : - n 0x6E 00004101 0x00001005 : - . 0x06 00004102 0x00001006 : - õ 0xF5 00004944 0x00001350 : + . 0x05 [root@devel3 Storable.patch]#

So you need to remove the 7 bytes I originally suggested and then add 0x05 at offset 4944 in the *patched* file. ie this it the right offset if you have already removed the 7 bytes previously noted. The hexedit should look like:

00000FD8: 72 73 74 07 00 00 00 73 65 73 73 69 6F 6E 06 F5 00 00late +st......count.....fi 00000FF0: 65 73 73 69 6F 6E 5F 63 6F 75 6E 74 0A 01 4C 04 00 00rst. +...session.........s 00001008: 03 0E 00 00 00 04 02 00 00 00 00 14 00 00 00 61 63 74essi +on_count..L....type. 00001338: 05 05 00 00 00 6C 61 62 65 6C 0A 0B 78 78 78 78 78 78XX<B +R>XXXXXXXX XXXXXXXXX 00001350: 00 00 00 69 64 0A 03 4E 46 4C 0B 00 00 00 6C 65 61 67.... +.label..xxxxxxxxxxx. 00001368: 72 0A 1B 66 6F 6F 74 62 61 6C 6C 3A 78 78 78 78 78 78...i +d..NFL....league_abb

With those edits in place we can resurrect the original data (I think ;-) However it is fairly obvious that you have edited the binary to preserve the privacy of the user. I would just mention that you may well have corrupted the corruption :-)

[root@devel3 Storable.patch]# cat test.pl #!/usr/bin/perl use Storable qw(retrieve); $Storable::DEBUGME = 0; my $data = retrieve('store.patch'); use Data::Dumper; print Dumper $data; [root@devel3 Storable.patch]# ./test.pl $VAR1 = bless( { '_contractual' => bless( { '_VERSION' => 1, 'prefs' => { 'team/NHL' => + bless( { + 'category' => 'team/NHL', + 'favorite' => {}, + 'preferred' => {}, + 'hated' => {} + }, 'Preference' ), 'player/NBA' +=> bless( { + 'category' => 'player/NBA' + }, 'Preference' ), 'team/NCAAB' +=> bless( { + 'category' => 'team/NCAAB' + }, 'Preference' ), 'team/NCAAF' +=> bless( { + 'category' => 'team/NCAAF' + }, 'Preference' ), 'player/TEN' +=> bless( { + 'category' => 'player/TEN' + }, 'Preference' ), 'player/NCAAB +' => bless( { + 'category' => 'player/NCAAB' + }, 'Preference' ), 'team/NBA' => + bless( { + 'category' => 'team/NBA' + }, 'Preference' ), 'messages' => + { + 'last_post' => 1088213270, + 'latest' => [ + 'fantasybaseball|Fantasy Baseball Message Board|0000 +000000', + 'fantasybaseball|Fantasy Baseball Message Board|0000 +000000', + 'fantasybaseball|Fantasy Baseball Message Board|0000 +000000', + 'fantasybaseball|Fantasy Baseball Message Board|0000 +000000', + 'fantasybaseball|Fantasy Baseball Message Board|0000 +000000', + 'fantasybaseball|Fantasy Baseball Message Board|0000 +000000', + 'fantasybaseball|Fantasy Baseball Message Board|0000 +000000', + 'fantasybaseball|Fantasy Baseball Message Board|0000 +000000', + 'fantasybaseball|Fantasy Baseball Message Board|0000 +000000', + 'fantasybaseball|Fantasy Baseball Message Board|0000 +000000' + ] + }, 'player/AUTO' + => bless( { + 'category' => 'player/AUTO' + }, 'Preference' ), 'player/NCAAF +' => bless( { + 'category' => 'player/NCAAF' + }, 'Preference' ), 'team/AUTO' = +> bless( { + 'category' => 'team/AUTO' + }, 'Preference' ), 'component' = +> {}, 'team/TEN' => + bless( { + 'category' => 'team/TEN' + }, 'Preference' ), 'team/AUTORAC +ING' => bless( { + 'category' => 'team/AUTORACING' + }, 'Preference' ), 'player/PGA' +=> bless( { + 'category' => 'player/PGA' + }, 'Preference' ), 'player/MLB' +=> bless( { + 'category' => 'player/MLB', + 'preferred' => {}, + 'favorite' => {}, + 'basis' => 'explicit', + 'hated' => {}, + 'fantasy' => { + '00000' => 'explicit', + '0000' => 'explicit', + '000000' => 'explicit' + } + }, 'Preference' ), 'player/NFL' +=> bless( { + 'category' => 'player/NFL', + 'basis' => 'explicit', + 'fantasy' => { + '000' => 'explicit', + '0000' => 'explicit', + '00000' => 'explicit', + '000000' => 'explicit' + } + }, 'Preference' ), 'player/NHL' +=> bless( { + 'category' => 'player/NHL' + }, 'Preference' ), 'ui' => bless +( {}, 'UiPreference' ), 'team/PGA' => + bless( { + 'category' => 'team/PGA' + }, 'Preference' ), 'player/mayhe +m' => bless( { + 'category' => 'player/mayhem' + }, 'Preference' ), 'team/OLY' => + bless( { + 'category' => 'team/OLY' + }, 'Preference' ), 'team/GOLF' = +> bless( { + 'category' => 'team/GOLF' + }, 'Preference' ), 'team/MLB' => + bless( { + 'category' => 'team/MLB', + 'preferred' => {}, + 'favorite' => {}, + 'hated' => {} + }, 'Preference' ), 'team/TENNIS' + => bless( { + 'category' => 'team/TENNIS' + }, 'Preference' ), 'team/NFL' => + bless( { + 'category' => 'team/NFL', + 'favorite' => { + 'NONE' => 'explicit' + }, + 'basis' => 'explicit' + }, 'Preference' ) } }, 'ContractualData' ), '_hash_bucket' => 645, '_sequence' => '00', 'prefs' => { 'ui' => {} }, '_version' => 7, '_timestamp' => 1089225294, 'registration' => bless( { 'country' => 'USA', 'modifieddate' => 'XX/XX/X +XXX', 'mi_id' => '0000000', 'sportsclub_id' => '-00000 +00000000000000', 'firstname' => 'XXXX', 'phone' => 'XXXXXXXXXX', 'state' => 'MD', 'lastname' => 'XXXXXXXX', 'address1' => 'XXXX XXXXXX +XXX XXXXXXXX XXXX', 'address2' => '', 'birthdate' => 'XX/XX/XXXX +', 'city' => 'XXXXXXXX XXXX', 'gender' => '', 'timezone' => 'EST', 'occ_code' => '', 'zip' => '00000-0000', 'rewards' => '0', 'email' => 'xxxxxxxxx@xxxx +xxxxx.xxx', 'products' => { '00' => { +'status' => 1, +'prod_id' => '00' }, '000' => { + 'status' => 1, + 'prod_id' => '000' } +, '0000' => +{ + 'status' => 1, + 'prod_id' => '0000' +} } }, 'RegistrationInfo' ), 'id' => 'xxxxxxxxxxx', 'session_count' => 245, 'session' => { 'latest' => 1095301660, 'first' => undef, 'count' => 3 }, 'type' => 'L', 'fantasy' => { 'active_roster_mayhem' => [], 'teams_set' => '0000000000', 'active_team_type_MLB' => '_set', 'active_roster_MLB' => [ '000000', '00000', '00000', '000000', '00000', '000000', '0000', '000000', '000000', '0000', '00000', '000000', '000000', '0000', '0000', '0000', '000000', '0000', '000000', '0000', '000000', '0000', '0000', '000000', '00000', '000000', '00000', '000000', '0000', '0000', '0000', '00000', '0000' ], 'active_team_type_NFL' => '_set', 'active_roster_NFL' => [ '0000', '0000', '0000', '0000', '0000', '000000', '000', '0000', '0000', '000000', '000000', '000000', '000000', '000000', '000000', '0000', '00000', '000', '0000' ], 'active_roster_date_MLB' => '00000000' +, 'active_roster_date_NFL' => '00000000' +, 'teams' => [ { 'args' => 'pid=xxxxxxxx +xxx', 'base' => 'http://xxxxx +xxxxxxxxxxxxx.football.sportsline.com/', 'id' => 'xxxxxxxxxxx', 'label' => 'XXXXXXX XXX +XX XXX<BR>XXXXXXXX XXXXXXXXX', 'key' => 'football:xxxx +xxxxxxxxxxxxxx', 'league_abbr' => 'NFL', 'logo' => 'http://xxxxx +xxxxxxxxxxxxx.football.sportsline.com/ownerlogo?size=00x00' }, { 'args' => 'pid=xxxxxxxx +xxx', 'base' => 'http://xxxxx +.baseball.sportsline.com/', 'id' => 'xxxxxxxxxxx', 'label' => 'XXXXXXX XXX +XX<BR>XXX XXXXXXX XXXXXX', 'key' => 'baseball:xxxx +x', 'league_abbr' => 'MLB', 'logo' => 'http://xxxxx +.baseball.sportsline.com/ownerlogo?size=00x00' }, { 'args' => 'pid=xxxxxxxx +xxx', 'base' => 'http://xxxxx +xxxxxxxxxxx.2.mayhem.sportsline.com/', 'id' => 'xxxxxxxxxxx', 'label' => 'XXXX XXXXXX +XX<BR>CBS SportsLine.com Round by Round Challenge', 'key' => 'mayhem:xxxxxx +xxxxxxxxxx.2', 'league_abbr' => 'mayhe +m', 'logo' => 'http://xxxxx +xxxxxxxxxxx.2.mayhem.sportsline.com/ownerlogo?size=00x00' }, { 'args' => 'pid=xxxxxxxx +xxx', 'base' => 'http://xxxxx +xxxxxxxxxxxxxxxx.2.mayhem.sportsline.com/', 'id' => 'xxxxxxxxxxx', 'label' => 'XXXX XXXXXX +XX <BR>CBS SportsLine.com Perfect Bracket Challenge', 'key' => 'mayhem:xxxxxx +xxxxxxxxxxxxxxx.2', 'league_abbr' => 'mayhe +m', 'logo' => 'http://xxxxx +xxxxxxxxxxxxxxxx.2.mayhem.sportsline.com/ownerlogo?size=00x00' } ], 'active_roster_date_mayhem' => '000000 +00', 'active_team_MLB' => 'baseball:xxxxx', 'active_team_NFL' => 'football:xxxxxxx +xxxxxxxxxxx', 'active_team_type_mayhem' => '_set', 'active_team_mayhem' => 'mayhem:xxxxxx +xxxxxxxxxx.2' } }, 'Profile' ); [root@devel3 Storable.patch]#

cheers

tachyon

Replies are listed 'Best First'.
Re^2: Small Corrupted Storable Retrieve Dies With an Invalid "Out of memory!"
by jbisbee (Pilgrim) on Oct 27, 2004 at 16:54 UTC
    I've applied your binary patch and resurrected the original storable file so I know my privacy string edits (all the 'xxx..' and '000..' didn't have any affect on the resulting edited storable that I posted.

    I've pulled Storable 1.0.13 off of backpan and am currently preparing a to apply your "Bugger" patch. Thank you very much for you indepth analysis and I'll be sure to follow up if I can record a dump of the data structure that caused it in the first place. (that is if I can reproduce the corrupt storable with multiple attempts with Storable 1.0.13 and 2.13)

    I'll follow up on this thread when/if I can determine the actual cause. :P

    Thanks again!

    -biz-

      It was very thoughtful of you to edit the data. ++ for that. I presume you worked your way through it with a hex editor as I don't see how you could have done it with a regex. If you did manage to do it with REs please explain ;-)

        I edited the storable with... vim!!!! :P I actually did a range search and replace with vim and it worked like a charm. When looking at the file within vim, you can see hints of the data structure (scalars, arrays, and hashes). Along with running 'strings' against it, I was able to see what might be sensitive data and my intent was to censor it while keeping a storable that would reproduce the error.

        The lesson I learned was that I needed to keep the string length the exact same as the orginal file (which now makes perfect sense knowing that there are read length markers before each string).

        Before a Storable was a kind of mystical thing that just worked, and its really strange (for me anyway) to now be able to "see" and somewhat understand the file in a hex editor.

        I also applied your "patch" with vim as well, I removed the 7 characters you suggested and found another 0x05 entry in the file and yanked an pasted it in place. Then I ran my test case code and the damn thing thawed :) (I've did attempt to use an hex editor on my power book with little success because I've never really used one before). Anyway, vim++ :P

        -biz-

      The interesting thing is that with the edits the restored data structure store/retrieves just fine! Forgot to mention that. There must be an extra byte or two in the original that is triggering the behaviour but where....and what.....

      Without a test case data structure it is like looking for a very small haystack in an enormous pile of needles! Did I mention I did not like the author's macro mania much yet ;-) As soon as you have a test case (and the croak will get you one provided you test and save it) it should be relatively easy to track down the issue(s)

      Storable is a module I use all the time, as do many others. We can all do without it randomly crashing. Good luck.

      cheers

      tachyon

Re^2: Small Corrupted Storable Retrieve Dies With an Invalid "Out of memory!"
by husker (Chaplain) on Oct 27, 2004 at 15:35 UTC
    Wonderful analysis, tachyon. You are an Example and Inspiration to Us All!
      Hey, saw you nick figured it was a football reference. I actually grew up in Omaha and attended school at the University of Nebraska at Omaha.

      Go Big Red! :)

      -biz-