Endianness also applies in the numbering of bits within a byte or word. In a consistently big-endian architecture the bits in the word are numbered from the left, bit zero being the most significant bit and bit 7 being the least significant bit in a byte.
So endianness should have to do with bit order. Just because the usual way of packing/unpacking little- or big-endian data in Perl (Network and VAX types) does not follow this pattern it doesn't mean it's not correct.