Re: Random templating

Dear sulfericacid,

Many people are warning you that randomly switching the layout and look-and-feel of your site is likely to harm your site's usability. However, only you are in a position to make that call. Maybe you know something that we don't. Maybe your site is all about art and exploration, or maybe your users come to your site for an "experience" and not to find information, do their jobs, or purchase a product. In any case, it's your call to make.

What's particularly interesting about your situation is that it provides a perfect opportunity for all of us to learn something. Is the common wisdom – that swapping UIs frequently is "bad" – applicable to your site?

Let's find out!

Let's design a simple experiment to measure the effect of random UI swapping on your users:

Partition your users into two equally sized groups: the experimental group (those who will be subjected to random UI swapping every time they log in) and the control group (those who won't). To eliminate bias among the UIs, each user in the control group should be randomly assigned one of the two new UIs, which then becomes that user's UI for the duration of the study.
Whenever users log in, determine which group they are in and select their UIs accordingly.
After a month or so, analyze your logs, site feedback, etc. to determine if there is a significant difference in behavior between the experimental and control groups. (For example, if common wisdom is correct, we would expect that site usage among experimental-group users to drop off w.r.t. control-group users.)

Sounds like fun, don't you think? When it comes to your site, care to prove the experts wrong? (Or right?)

Cheers,
Tom

Update: clarified wording.

Tom Moertel : Blog / Talks / CPAN / LectroTest / PXSL / Coffee / Movie Rating Decoder

Comment on Re: Random templating

Replies are listed 'Best First'.
Re^2: Random templating by Your Mother (Archbishop) on Oct 09, 2004 at 17:25 UTC
I agree the proof is in the pudding and benchmarking is crucial to having valid opinions but I've seen the pudding served to 10-20 million users a day all year and even minor changes like moving a linked heading lower or higher relative to other page blocks does change usage. This isn't really an experimental problem any more. What if someone moved your mousepad every morning? Oh, that's amusing. As discussed, a pure design site or something might make constant change interesting or desirable, otherwise it's probably a vanity choice and not really in your users' good interests. And even the design sites that do this kind of thing stay strongly branded with a big headline/logo/etc. I can't think of a single site I've visited more than once in 10 years that does this though. On the other hand, letting users select a template or stylesheet is most definitely in their interest and good service. And if it's your site, it's your call. The other problem with doing it via two different code bases instead of CSS or templates is you just locked yourself into a future course if you intend to keep users who do like the different systems happy. If you upgrade, redraft, change then your costs/time just doubled. If you can change one of the systems to run with the template of the other (the end product of HTML doesn't care what created it) you'll probably be happier as time goes by. Plus you're in an unusual and enviable position. You have two code bases done and running, and you could pick the better written of them.	[reply]

Replies are listed 'Best First'.

Re^2: Random templating
by Your Mother (Archbishop) on Oct 09, 2004 at 17:25 UTC

I agree the proof is in the pudding and benchmarking is crucial to having valid opinions but I've seen the pudding served to 10-20 million users a day all year and even minor changes like moving a linked heading lower or higher relative to other page blocks does change usage. This isn't really an experimental problem any more. What if someone moved your mousepad every morning? Oh, that's amusing.

As discussed, a pure design site or something might make constant change interesting or desirable, otherwise it's probably a vanity choice and not really in your users' good interests. And even the design sites that do this kind of thing stay strongly branded with a big headline/logo/etc. I can't think of a single site I've visited more than once in 10 years that does this though. On the other hand, letting users select a template or stylesheet is most definitely in their interest and good service. And if it's your site, it's your call.

The other problem with doing it via two different code bases instead of CSS or templates is you just locked yourself into a future course if you intend to keep users who do like the different systems happy. If you upgrade, redraft, change then your costs/time just doubled. If you can change one of the systems to run with the template of the other (the end product of HTML doesn't care what created it) you'll probably be happier as time goes by. Plus you're in an unusual and enviable position. You have two code bases done and running, and you could pick the better written of them.

[reply]