In a dedicated benchmark to test this, binmode doesn't change anything at all:

with a binmode call:

            Rate with_rs without
with_rs 254857/s      --    -34%
without 384458/s     51%      --

without a binmode call:

            Rate with_rs without
with_rs 260564/s      --    -31%
without 375127/s     44%      --
[download]

I've just posted the question to the perl5 porters.