i use ssh -X for that.

Yes, that would work, if set up with ssh keys. I typically create adminstrative users like the one I'm talking about without any passwords or ssh keys. It removes a possible line of attack. (Yes, I'm paranoid).