hmbscully has asked for the wisdom of the Perl Monks concerning the following question:

In April 2006, I asked for the wisdom of the monks regarding encryption and file locking. Fast-forward to the present and I am begrudgingly being forced to implement the previously described problem. For a change, I am working with another programmer on this, though neither of us have any experience with encryption before tackling this project.

Code background story
The script takes form input, evaluates it based on a lot of factors, and then builds a fixed-length (1850 characters) line for each submission and writes that data into a flatfile, which is then passed off to another system for processing. Each flatfile can end up with anywhere from a few dozen to a few thousand lines per day.

Here's an example of how the code currently works (snipping out the long section of building many, many variables:

#!/usr/bin/perl use strict; use warnings; use CGI qw(:standard); use CGI::Carp qw(fatalsToBrowser); use Fcntl ":flock"; use POSIX; use File::CounterFile; use Crypt::Rijndael; use Crypt::CBC; require '../code_paths.conf'; [a lot of variable declarations go here] #SCRIPT setValues(); writeSubmissions(); exit; #SUBROUTINES sub setValues { [a lot of values are read in and set and worked with] #here's where we build the line # format for fixed-length line: my $filebegin_format = "%-1.1s%-8.8s%6.6d%-17.17s%-25.25s%-16.16s% +-1.1s%-9.9s%-9.9s%-8.8s%-35.35s%-25.25s%-2.2s%-12.12s%-3.3s"; #file l +ayout for characters 1-177 my $request_format = "%-4.4s%-4.4s%-1.1s%-1.1s%-151.151s"; #file l +ayout for requests 1-8 my $request9_format = "%-6.6s%-4.4s%-1.1s%-1.1s%-1.1s%-35.35s%-35. +35s%-35.35s%-25.25s%-2.2s%-12.12s%-3.3s%-4.4s"; #file layout for requ +est 9 my $fileend_format = "%-5.5d%-1.1s%-16.16s%-4.4s%-25.25s%-16.16s%- +9.9s%-1.1s%-40.40s%-14.14s%-2.2s%-9.9s%-3.3s%-3.3s%-10.10s%-40.40s%-2 +2.22s%-1.1s"; #file layout for characters 1630-1850 my $format = $filebegin_format . $request_format . $request_format + . $request_format . $request_format . $request_format . $request_for +mat . $request_format . $request_format . $request9_format . $fileend +_format; # build fixed-length line: $fl_line = sprintf $format, $filetype, $batch, $serial_number, $sy +s_time, $tt_lastname, $tt_firstname, $tt_mi, $ssn, $ssn_cor, $dob, $t +t_address, $tt_city, $tt_state, $tt_zip, $tt_country, $code_1, $date_ +1, $type_1, $priority_1, $filler, $code_2, $date_2, $type_2, $priorit +y_2, $filler, $code_3, $date_3, $type_3, $priority_3, $filler, $code_ +4, $date_4, $type_4, $priority_4, $filler, $code_5, $date_5, $type_5, + $priority_5, $filler, $code_6, $date_6, $type_6, $priority_6, $fille +r, $code_7, $date_7, $type_7, $priority_7, $filler, $code_8, $date_8, + $type_8, $priority_8, $filler, $code_9, $date_9, $type_9, $exception +_9, $priority_9, $addressee_9, $institution_9, $address_9, $city_9, $ +state_9, $zip_9, $country_9, $filler, $bill_total, $cc_type, $cc_numb +er, $cc_exp, $cc_name, $cur_lastname, $cur_firstname, $cur_mi, $cur_a +ddress, $cur_city, $cur_state, $cur_zip9, $cur_pcode, $cur_country, $ +phone, $email, $filler, $reject; $asr_line = uc($fl_line); #uppercase $asr_line =~ tr/\000-\037/ /; # removing control characters, be +st option for speed } #subroutine to keep writing the plain file sub writeSubmissions{ if ($filetype == 1 ) { $log_file = "pem.txt"; } else { $log_file = "act.txt"; } open LO, ">>$log_file"; flock LO, LOCK_EX; print LO "$asr_line\n"; close LO; }

Adding encryption
After many conversations about security and encryption, we decided we would encrypt each line before it gets written into the file, so the submitted data would never be anywhere but memory unencrypted. Then, after the file was passed onto another server in a different secure system, that system would deal with opening the file, decrypting each line, writing the decrypted data into a file for processing (don't ask why, its complicated).

We're attempting to do both the encrypting and decrypting on the same server during development since obviously we need to be able to decrypt the file to confirm we are getting the encryption correct.

The current attempt is to replace the writeSubmissions() subroutine with this code:

sub writeSubmissions{ if ($filetype == 1 ) { $log_file = "pem.enc"; } else { $log_file = "act_encrypted.enc"; } $my_key = `openssl enc -bf-cbc -d -in key_file.txt -k encrypt`; $cipher = Crypt::CBC->new( {'key' => $my_key, 'cipher' => 'Rijndael', 'regenerate_key' => 1, 'padding' => 'standard', }); open FH_crypted, ">>$log_file"; #binmode FH_crypted; flock FH_crypted, LOCK_EX; $cipher->start('encrypting'); print FH_crypted $cipher->crypt($asr_line); print FH_crypted $cipher->finish; close FH_crypted; }
and use this script to decrypt a command line-specified file
#!/usr/bin/perl use strict; use warnings; use Crypt::Rijndael; use Crypt::CBC; #-------------------------------------------------------------------- # Parameters #-------------------------------------------------------------------- my $my_key; my $plain_text; my $encrypted = $ARGV[0]; my $cipher; my $buffer; my $decrypted = $encrypted; $decrypted =~ s/\.enc/\.txt/; $my_key = `openssl enc -bf-cbc -d -in key_file.txt -k encrypt`; $cipher = Crypt::CBC->new( {'key' => $my_key, 'cipher' => 'Rijndael', 'regenerate_key' => 1, 'padding' => 'standard', }); #-------------------------------------------------------------------- # Decryption #-------------------------------------------------------------------- open(FH_encrypted, "<$encrypted"); open(FH_decrypted, ">$decrypted"); $cipher->start('decrypting'); while (read(FH_encrypted,$buffer,1851)) { print FH_decrypted $cipher->crypt($buffer); } print FH_decrypted $cipher->finish; close FH_encrypted; close FH_decrypted;

Which isn't working. I'm writing the line both into an encrypted and a plain text version while testing (for comparisons) and the plain/original version file is 1851 bytes while the encrypted version is 1872 bytes (which I assume is padding for the blocksize stuff that I don't really get). When we run the decrypt script the file is then 1856 bytes (and is still unreadable garbly-gook, but different garbly-gook than the original encrypted file.)

I realize there are many factors in dealing with encryption, so I'm looking to see if there is something inherently flawed in the logic of our current plan. One other thing to note that I'm trying to figure out is that the html page with the form and the cgi script it calls, as well as the decrypt script are all owned by a user "webmaster" but the file that the script writes is owned by "apache". Could this be a source of the problem? I'm trying to figure it out but I don't have the permissions on this server to change users or ownership and I'm trying to figure out what to communicate to my unix admin that should (if anything) be changed.

Thanks!


I learn more and more about less and less until eventually I know everything about nothing.

Replies are listed 'Best First'.
Re: line by line Encryption fun with Crypt::CBC and Rijndael? File Ownership issues?
by moritz (Cardinal) on Dec 03, 2007 at 21:53 UTC
    I haven't looked into the details of your script, but I don't think the "encrypt line by line in CBC mode" is likely to work.

    Block cyphers like Rijndael work with fixed length blocks, usually with block length = key length.

    If you want to use CBC, you should take care that all but the last block have a length that is a multiple of the key length (actually I don't know if that's really a requirement, but I can imagine a hundred reasons why it might blow up if you don't honor that constraint).

    It seems that you start the CBC mode anew for every line that you write (in the encryption), but try to decrypt the whole file at once in CBC mode. That won't work, you have to take the exact steps in reverse (in this case reset the cypher block chain after each encrypted line. Which kinda defeats the CBC idea). But since the encrypted text may contain newlines, you can't rely on line logic anymore - you'll have to invent a binary format for that.

    As for the potential permission issue: just test the encryption/decryption part on the command line, preferably first in memory, i.e. without writing the data to the file.

    Update: a bit more about CBC: symmetric, block based cyphers suffer from the weakness that if you encrypt large amounts of data, a potential attacker can gather much data that is all encrypted with the same key. Additionally if a block of identical data recurs multiple times, it will result in identical encrypted blocks, thus revealing information on an eavesdropper.

    To circumvent these problems the "Cypher Block Chaining" was invented. That's a protocol that defines a way that the encryption is changed based on prior encoded blocks.

    More exactly: the plain text of the current block is XOR'ed with the cypher text of the previous block.

    Which means that for every block (except the first one) you not only need the key to encrypt it, but also the preceding block.

    So if you want to decrypt CBC data you need reverse the encryption process exactly.

    (The fine print: I'm not a crypto guru, so please all correct me if I wrote bullshit ;)

      If you want to use CBC, you should take care that all but the last block have a length that is a multiple of the key length

      No. The blocks passed to crypt have no relation to cypher blocks. The extra bytes will be buffered until the next call to crypt.

      The start/crypt/finish mode allows you encrypt and decrypt arbitrary segments of the message (file) at a time, as long as you process the entire message from the start.

      In fact, encrypt and decrypt are just thin wrappers around start/crypt/finish.

      To circumvent these problems the "Cypher Block Chaining" was invented. [...] Which means that for every block (except the first one) you not only need the key to encrypt it, but also the preceding block.

      Right, although Crypt::CBC normally uses a special value for the first block too: the salt.
      Crypt::CBC uses salting to ensure that every message is encrypted with a different key.
      Crypt::CBC uses chaining to ensure that every block is encrypted with a different key.

      So if you want to decrypt CBC data you need reverse the encryption process exactly.

      Aye, and he doesn't do that. If he really did want to add a line at a time, he'd file would have to look like

      length-of-encrypted-line, encrypted-line, length-of-encrypted-line, en +crypted-line, ...

      Each line would be a message that would be encrypted, and then decrypted individually (using encrypt and decrypt).

        I wrote the code you need.

        Some utility functions

        sub read_bytes { my ($fh, $to_read) = @_; my $buf = ''; while ($to_read) { my $bytes_read = read($fh, $buf, $to_read, length($buf)); die("$!\n") if !defined($bytes_read); die("Unexpected end of file\n") if !$bytes_read; $to_read -= $bytes_read; } return $buf; } sub read_uint32 { my ($fh) = @_; return (unpack('N', read_bytes($fh, 4))); } sub read_str { my ($fh) = @_; my $length = read_uint32($fh); return read_bytes($fh, $length); } sub write_uint32 { my ($fh, $n) = @_; print $fh (pack('N', $n)); } sub write_str { my ($fh, $str) = @_; print $fh (pack('N', length($str)), $str); }

        The line-by-line encrypter:

        sub writeSubmissions { my ($cipher, $log_file, $str) = @_; open(my $FH_encrypted, '>>', $log_file) or die; binmode $FH_encrypted; flock $FH_encrypted, LOCK_EX; write_str($FH_encrypted, $cipher->encrypt($str)); } my $key_file = 'key_file.txt'; my $act_log_file = "pem.enc"; my $pem_log_file = "act_encrypted.enc"; my $my_key = `openssl enc -bf-cbc -d -in \Q$key_file\E -k encrypt`; my $cipher = Crypt::CBC->new({ key => $my_key, cipher => 'Rijndael' }) +; writeSubmissions($cipher, ..., ...);

        The line-by-line decrypter:

        my $key_file = 'key_file.txt'; my $encrypted = '...'; my $decrypted = '...'; my $my_key = `openssl enc -bf-cbc -d -in \Q$key_file\E -k encrypt`; my $cipher = Crypt::CBC->new({ key => $my_key, cipher => 'Rijndael' }) +; open(my $FH_decrypted, '>', $decrypted) or die("Unable to create decrypted file \"$decrypted\": $!\n"); binmode $FH_decrypted; flock $FH_decrypted, LOCK_EX; open(my $FH_encrypted, '<', $encrypted) or die("Unable to open encrypted file \"$encrypted\": $!\n"); binmode $FH_encrypted; flock $FH_encrypted, LOCK_SH; while (!eof($FH_encrypted)) { print $FH_decrypted $cipher->decrypt(read_str($FH_encrypted)); }

        Update: Fixed bugs mentioned in replies.
        Update: Tested (minus the openssl bit). Fixed other bugs.

Re: line by line Encryption fun with Crypt::CBC and Rijndael? File Ownership issues?
by quester (Vicar) on Dec 04, 2007 at 09:06 UTC
    You will have to restart the decryption process for each block of data. The encrypted data consists of the constant "Salted__" followed by an eight byte salt value. You can't concatenate separately encrypted blocks together because those headers are treated very differently than regular data.

    But in fact, it's only if you are on a fairly recent version of Crypt::CBC that you will have headers that start with "Salted__". Otherwise...

    ... otherwise take a large aspirin and ponder this ominous note in the documentation for Crypt::CBC version 2.22:

    IMPORTANT NOTE: Versions of this module prior to 2.17 were incorrectly using 8-byte IVs when generating the "randomiv" style of header, even when the chosen cipher's blocksize was greater than 8 bytes. This primarily affects the Rijndael algorithm. Such encrypted data streams were not secure. (emphasis the author's)
    My humble opinion is that openssl would be a somewhat safer choice in terms of maturity and the number of eyes on the code. Crypto modules in general are notorious for subtle lurking exotic bugs that have no effect on anything, other than making your data much easier to steal than you would like to think. That was true centuries ago, it was true when the Enigma was invented, and it still seems to be true today.

    If you think that Crypt::CBC is a good choice, consider this: Crypt::CBC with Rijndael is supposed to be the same as openssl enc -aes-128-cbc. Or is is -aes-192-cbc? Or -aes-256-cbc? (I checked and it's actually -aes-256-cbc, but I couldn't actually find anywhere that said that in the documentation.)

    If you really want to use Crypt::CBC you at least want to go with the very latest version. Be positive that /dev/urandom exists and is known to work on your kernel. (If it doesn't Crypt::CBC will fall back on the Perl built-in random number generator, which is a Very Bad Thing cryptologically.) And keep a sharp eye out for later releases that might have more bug reports in them....

    (and ditto on the disclaimer: I'm not a crypto guru either!)

      We had seen that ominous note and we have version 2.24 of Crypt::CBC.

      The reason I'm using Crypt::CBC is because it was suggested to me somewhere previously when I was pondering encryption. I honestly don't understand this stuff as much as I should. We originally looked at openSSL, but then I was told I had to use the AES standard which, after investigation, we thought was the Rijndael implementation.

      I'm going to take the fairly large aspirin now...


      I learn more and more about less and less until eventually I know everything about nothing.
        You could use openssl enc -aes-256-cbc. That's probably the best compliance with the notion of "standard" that you could find.

        Be careful not to pass the encryption key as an environment variable, since the environment variables can be visible in ps in most Unix-like operating systems. And do double check that you have a usable /dev/urandom.

Re: line by line Encryption fun with Crypt::CBC and Rijndael? File Ownership issues?
by ikegami (Patriarch) on Dec 05, 2007 at 17:36 UTC

    In both programs, you should binmode on the fh from which you read and on the fh to which you write. It makes a difference on some platforms, and won't harm anything on the others.

    Also, flock FH_crypted, LOCK_EX; is useless unless you also use flock FH_crypted, LOCK_SH; in the decryptor.

Re: line by line Encryption fun with Crypt::CBC and Rijndael? File Ownership issues?
by girarde (Hermit) on Dec 05, 2007 at 14:52 UTC
    This is a good post, and would have been REALLY good with the use of <readmore> tags
      I attempted to use the readmore tags, but I had them placed incorrectly. I have fixed this.

      I learn more and more about less and less until eventually I know everything about nothing.