No, there's nothing to worry about ( when doing floating point operations on < 52 bit ints stored in doubles ). Floating point errors come from the inability to represent a number accurately within the available precision. No error is loss when the number can be accurately represented within the available precision.

Update: Added the text in bold parens for clarity.