Or, so that they don't feel left out, pack and unpack:

  use MIME::Base64 qw( decode_base64 );

  my $scrambled = decode_base64($_);
  my @scrambled = unpack('C*', $scrambled) ;
  my @orig      = map { $scrambled[$_-1] ^ $scrambled[$_] } 1..$#scram
+bled;
  my $orig      = pack('C*', @orig) ;
[download]

In this case the magic byte-wise string xor (^) is clearly the best bet. But in general I'd use pack/unpack to do string <-> byte array -- and, of course, so much more.

However, watch out for utf8 strings... unpack('C*', ..) will unpack a utf8 string as characters, so may return values > 0xFF -- so:

  my $b = "123\xC4" ;
  my $u = "a\xC4\x{107}\x{1C4}" ;

  show(unpack('C*',   $b)) ;  # 0x31, 0x32, 0x33, 0xC4
  show(unpack('C*',   $u)) ;  # 0x61, 0xC4, 0x107, 0x1C4

  { use bytes ;
    show(unpack('C*', $b)) ;  # 0x31, 0x32, 0x33, 0xC4
    show(unpack('C*', $u)) ;  # 0x61, 0xC3, 0x84, 0xC4, 0x87, 0xC7, 0x
+84
  } ;

  sub show {
    print join(", ", map(sprintf("0x%02X", $_), @_)), "\n" ;
  } ;
[download]

as shown, use bytes causes the unpack to operate on the bytes of encoded form of the utf8 string.

Note that with utf8 strings, pack('C*', ... and unpack('C*', ... are not symmetrical, observe:

  my $u = "a\xC4\x{107}\x{1C4}" ;

  my @u = unpack('C*', $u) ;
  show(@u) ;                  # 0x61, 0xC4, 0x107, 0x1C4
  bytes(pack('C*',@u)) ;      # Character in 'C' format wrapped in pac
+k ...
                              # Character in 'C' format wrapped in pac
+k ...
                              # 61:C4:07:C4 -- byte

  sub show {
    print join(", ", map(sprintf("0x%02X", $_), @_)), "\n" ;
  } ;

  sub bytes {
    my ($s) = @_ ;
    my $w = utf8::is_utf8($s) ? "utf8" : "byte" ;
    use bytes ;
    print join(":", map(sprintf("%02X", $_), unpack('C*', $s))), " -- 
+$w\n" ;
  } ;
[download]

but this is wandering off topic into a significantly murky area, of possibly marginal utility...