Well, you can simplify things greatly by working with two items at a time. But that just forces more complexity into the caller:

sub c2_out {
    my( $fh, $x, $y )= @_;
    if(  defined $y  ) {
        print $fh pack "C3",
            $x & 0xff, ($x>>8) | ($y&0xf)<<4, $y>>4;
    } else {
        print $fh pack "CC", $x & 0xff, $x>>8;
    }
}

sub c2_in {
    my( $fh )= @_;
    my $len= read( $fh, $c, 3 );
    return   if  ! $len;
    die "Extra trailing byte (",unpack("C",$c),")"
        if  1 == $len;
    my( $x, $y, $z )= unpack "C*", $c;
    $x |= ($y&0xf)<<8;
    return $x   if  ! defined $z;
    return( $x, $z<<4 | $y>>4 );
}
[download]

Or, dropping the file handle support:

{
    my @nybbles;
    sub c_out {
        push @nybbles, 0   if  ! @_;
        @_= ( @nybbles, map { $_&0xf }
            map { $_, $_>>4, $_>>8 } @_ );
        print OUT pack "C", shift|(shift()<<4)   while  1 < @_;
        @nybbles= @_;
    }
}

{
    my @n;
    sub c_in {
        do {
            return shift(@n) | shift(@n)<<4 | shift(@n)<<8
                if  2 < @n;
            push @n, map { $_&0xf, $_>>4 } unpack "C*", <IN>;
        } while(  1 < @n  );
        return;
    }
}
[download]

As usual, the emphasis seems to be on "sexy" code rather than "good" code. ):

Update: The above code has now been tested. I've come to like the map version and would probably modify it to make it robust and flexible (which would up the number of lines, against the point of this node) if I needed this functionality.

        - tye (but my friends call me "Tye")