Is binary data even allowed in emails?

Don't think so - if it were, we would not need Base64 encoding and all that tricky Mime business:)