Some more debugging discovered some embarrassing things.
Being so focused on the borked metadata, I did not realize that the image also got resized using Image::Magick. I gues the problem is not Image::ExifTool after all. Ill gues I now suspect Image::Magick :)
Thank you for all you help so far.
Best regards.