That's because File::Binary's code is wrong. Specifically, the seek() method is written as so:

sub seek {
    my $self = shift;
    my $seek = shift;
    unless ($self->{_is_seekable}) {
        carp "FH is not seekable" if $DEBUG; 
        return 0;
    }

    $self->{_fh}->seek($seek) if defined $seek;
    $self->_init_bits();
    return $self->{_fh}->tell();
}
[download]

Which is wrong. The whole module is written rather sloppily, frankly. Which is not surprising, given that the test suite doesn't even have a test for seek().

Furthermore, the test coverage for the 2874 tests is only

---------------------------- ------ ------ ------ ------ ------ ------
+ ------
File                           stmt   bran   cond    sub    pod   time
+  total
---------------------------- ------ ------ ------ ------ ------ ------
+ ------
blib/lib/File/Binary.pm        79.2   47.6   37.5   87.1   95.0  100.0
+   73.1
Total                          79.2   47.6   37.5   87.1   95.0  100.0
+   73.1
---------------------------- ------ ------ ------ ------ ------ ------
+ ------
[download]

If I were you, I would find some other way to do what you're trying to do.