I'm looking for gotchas, pointers, samples and other things I can use to learn how to do this.

Some things I can recommend from experience:

• don't do the XmlHttpRequest's yourself (use a mature, robust, cross-platform wrapper. eg: Dojo.io)
• don't bother with REST, you'll depend too much on the webserver accepting non-"standard" request types.
• do use JSON or SOAP (I'd say JSON as most browser SOAP implementations are not well done)
• do perform request validation on both the client and server side (on the client to help the user, on the server for security)
• do make sure to implement some form of sequencing in your session-management so that service replies aren't handled out-of-order by the client (if that's a problem)
• do handle exceptions on the server side gracefully, all client-side submission Javascript should be ready to handle an error condition (not related to HTTP)
• don't use synchronous AJAX calls under any circumstances as they have extremely poor robustness in most browser implementations.
• do make sure to use a tool like FireBug in order to debug (and profile) you're AJAX requests.
• do show the user some useful indication that the server is doing something; robustly handle error conditions so that the user-interface never hangs.
• if the user shouldn't double-click (or some other form field interaction), make sure they can't.
• if possible, detect old/phone-based browsers and redirect them to an "old school" form on a different page.

-David.