I think you've hit upon your own solution. Only include the Javascript that is completely optional. Rollovers are fine (I suppose), but do not rely on Javascript to be your only validation mechanism for CGI scripts (for example). (This is something that the Rat book mentions.)

I don't use Javascript to validate CGI scripts as the moment because my scripts are generally small enough that the performance hit isn't that bad. This also keep me from having to maintain two separate codes bases for the same task (validating input) and is helping my Perl skills.

As far as browsing goes, I generally have Javascript disabled--unless I know that a site has a Javascript-based feature that I need/like or unless the site has really good use of CSS. (Netscape 4.7x does not render CSS unless Javascript is enabled.)

Also, I use different browsers at different times. I use one for sites I trust or need Javascript/CSS with and I use another (with everything disabled) for most other activities. I've found this useful to test my sites with as well and generally ensure that my sites degrade as nicely as reasonably possible.

--f