Reading encoded file with Perl

a115764m has asked for the wisdom of the Perl Monks concerning the following question:

I have a password file that has some binary data & user name and passwords are readable in a hex editor or with this code

 
#!/usr/bin/perl -w
use strict; 
my $buff;
my $file ="/Perl/scripts/old/bak_PW/dkxy01_";
system("cls");
open (FH, $file ) or die "can't open $file: $!";
binmode(FH);
binmode(STDOUT); 
while (read(FH, $buff, 324)) {
    print STDOUT $buff;
}
close FH      or die "Closing: $!";
[download]

Below is output from a hex editor it seem like the data is in blocks of 324 bytes after the header, so from offset 0->23B is a header of sometype and the first record is from 23C->37F with user = MFG and password = MFG, next is from 380->4C3 user = ADRIANC and password = 08008 and it goes on from there.

55 73 65 72 4C 6F 67 69 6E 55 73 65 72 50 61 72    UserLoginUserPar
61 6D 2E 4F 55 4F 55 55 73 65 72 00 00 00 00 00    am.OUOUUser.....
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 30 30 2E 30    ............00.0
30 2E 30 31 00 00 00 00 00 00 00 00 00 00 00 00    0.01............
00 00 00 00 D5 18 D9 46 7C 4D DB 4D 00 00 00 00    ....Õ.ÙF|MÛM....
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
24 00 00 00 41 44 41 4D 20 55 50 44 41 54 45 20     $...ADAM UPDATE
35 2F 32 34 2F 32 30 31 31 00 00 00 00 00 00 00    5/24/2011.......
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 23 4B 45 20 48 6F 73 79    ........#KE Hosy
75 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    u...............
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 44 4B 58 59 30 31 00 00 00 00 00 00    ....DKXY01......
00 00 00 00 00 00 00 00 41 FB 01 00 0E E2 01 00    ........Aû...â..
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 4B 00 00 00 00 00 00 00 52 F8 7A C4    ....K.......RøzÄ
02 00 00 00 63 78 35 30 30 30 00 00 00 00 00 00    ....cx5000......
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 4D 46 47 00 00 00 00 00    ........MFG.....
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 4D 46 47 00    ............MFG.
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00    ................
CC FD AF C4 02 00 00 00 63 78 35 30 30 30 00 00    Ìý¯Ä....cx5000..
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 41 44 52 49    ............ADRI
41 4E 43 00 00 00 00 00 00 00 00 00 00 00 00 00    ANC.............
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
30 38 30 30 38 00 00 00 00 00 00 00 00 00 00 00    08008...........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 05 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
02 00 00 00 96 FA E0 EA 02 00 00 00 63 78 35 30    ....–úàê....cx50
30 30 00 00 00 00 00 00 00 00 00 00 00 00 00 00    00..............
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
44 41 4E 4E 59 57 00 00 00 00 00 00 00 00 00 00    DANNYW..........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 31 37 37 32 00 00 00 00 00 00 00 00    ....1772........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 03 00 00 00 96 FA E0 EA 02 00 00 00    ........–úàê....
63 78 35 30 30 30 00 00 00 00 00 00 00 00 00 00    cx5000..........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 54 45 44 4E 00 00 00 00 00 00 00 00    ....TEDN........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 38 35 36 35 00 00 00 00    ........8565....
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 04 00 00 00 82 52 55 E2    ............‚RUâ
02 00 00 00 63 78 35 30 30 30 00 00 00 00 00 00    ....cx5000......
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 41 44 41 4D 42 00 00 00    ........ADAMB...
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 35 33 33 38    ............5338
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00 00 00 00 00 00 00 00 00 00 00 00 05 00 00 00    ................
52 1B 8F 0A 03 00 00 00    R.......
[download]

my questions are how would one go about unpacking and repacking when one is unsure of what all the extra data means, is there an easy way of looking at this kind of a problem, I have read about intel,moto,big I and little I, but I am not sure if any of that applies here and if so how. My end goal is to make changes to the file and then upload it back to the tools so that all 65+ tools do not have to be manualy updated. Some of the records start with "RøzÄ" or "–úàê" and others ",RUâ". Also cx5000 seems to be an important stop or start point. Any ideas or links to indepth pack/unpack with detailed examples would be great if you are unable to id the system used to encode

Comment on Reading encoded file with Perl Select or Download Code

Replies are listed 'Best First'.
Re: Reading encoded file with Perl by JavaFan (Canon) on Aug 05, 2011 at 08:30 UTC
Without knowing the structure of the file, it's impossible to say what to do. Even if you could determine which fragments are passwords and which aren't, just replacing the passwords with something else may corrupt the file; there could be a checksum somewhere, and some of your other 65+ tools may no longer be able to open the file. Also, if you don't know the structure, you don't know whether a byte sequence is a string in some encoding, a numeric value in some encoding, or something else. Which makes it impossible to find an pack/unpack string for.	[reply]
Re: Reading encoded file with Perl by moritz (Cardinal) on Aug 05, 2011 at 08:55 UTC
If you have other tools that can understand this file format, try to make small changes to the file through these tools (change a password but keep it at the same length, change a password to one with a different length, change a username, add a user, delete a user, ...), and calculate the difference that it makes to the file. That should give you some idea what parts of the file stand for what. If some part of the file is always changed, it could be a checksum or a timestamp. If it's always increasing, it's most likely a timestamp, if it changes seemingly randomly, it's most likely a checksum. Perl 6 - second systems done right	[reply]
Re: Reading encoded file with Perl by graff (Chancellor) on Aug 06, 2011 at 02:36 UTC
My end goal is to make changes to the file and then upload it back to the tools so that all 65+ tools do not have to be manualy updated. If your goal is specifically to change the user name and/or password for one or more records, where the existing username and password are already known before you read the file, then you have a reasonable chance of success -- so long as you follow the advice in the 2nd reply above -- because it looks like those are fixed-width fields. Even if you don't have a clear idea of where the record boundaries are, the fact that a known username and password can be found at a specific distance from each other in the input data makes it pretty simple to splice in a new username and/or password -- unless (as suggested in the first reply) this change needs to be incorporated into some sort of checksum value elsewhere in the record. If some simple experiments with valid tools (to apply edits that create valid/usable versions of the file) show that you can just change the username and password without further ado, you could slurp the whole file into a scalar, and apply `s///` based on the known old and new username/password: `# assuming INFILE and OUTFILE are already open, # and string variables are already set... my $match = $old_username . "\x00"x(84-length($old_username)) . $old_password . "\x00"x(84-length($old_password)); my $replace = $new_username . "\x00"x(84-length($new_username)) . $new_password . "\x00"x(84-length($new_password)); $/ = undef; my $filedata = <INFILE>; $filedata =~ s/$match/$replace/; print OUTFILE $filedata;` [download] (updated to use the correct field width (84); actually, the password field width might be different -- no way to tell for sure, but 84 is probably a safe bet). If you're trying to change something else in the data besides user name and/or password, well, so far there just isn't enough information available to handle that. You need to get real documentation about the file format to build your own editor for it.	[reply] [d/l] [select]