I think you got your answer, but in terms of clarification, there are two sets of modes: user UMODE and channel MODE flags.

A client submits UMODE commands (or the server may assert UMODE changes) to adjust your client's relationship to the server or the network as a whole. These are typically not seen or manipulated by other clients at all, though server operators and ircd services like nickserv may have special abilities. User UMODE changes include things like invisibility from general queries, powers over service administration, flagging valid registered nicknames, etc.

Any client submits MODE commands only relative to a specific channel. The *other* participants in those channels will get a chance to witness the MODE adjustment; some servers assume that your client is smart enough to monitor the mode changes you submit on your own.

These MODE changes (like +o for chanops) are really referring to the state of the channel itself, not a part of your connection, and they just contain a reference to your nick or mask.