Can anyone recommend a good free guide on split-testing?

Ed

As Jason says, do. But you need to be assured that the results are statistically sound.

You need enough people actually being tested. It has to be statistically sound.

Basically, 'what the other guys said'.. but do talk to me before jumping in and using Google Optimizer - unless it has changed dramatically, there are better solutions out there.

Most split-test tutorials over simplify the significance testing and advise targeting .95 (eg 95% of being true), but depending on what you are testing, and why, there are different levels of statistical significance that may be acceptable. In general, we are not testing things that are of the same importance as a medical research, and we also have the option to keep testing pretty much indefinitely (ie an infinite sample size), which gives us more options and affects the decision process.

I have a few different software packages that will get the job done quicker than Google Optimiser, and I have spreadsheets that will calculate the statistical significance better too

ping me on Skype for more info

> Make sure the 2 pages are VASTLY different

Why?

Using exaggeration to prove my point turning the whole page upside down would probably upset the user and skew the test result. I would test a big change like that by simply asking users and then fine-tune using split-testing.

Dont forget our friends Karl and Ben:

http://www.conversion-rate-experts.com/articles/101-google-website-optimizer-tips/

Cheers for all the advice Jason, Rich et al.

Just something I'm looking into at the moment. Not actually ready to start doing it yet. When I do I'll be sure to hit you on Skype, Colin