I’m really looking forward to presenting on A/B Testing at the Net Tuesday event on December 4th and in preparation I’ve been compiling some of my thoughts and doing some additional research around the subject. That search led me to a webinar from Kiss Metrics (follow their blog!) and here were some of the key points.

3 Step Process of Good A/B Testing

  1. Insights. What have you discovered about your audience that makes you want to test something?
  2. Predictions. What do you expect will happen or would happen if you ran a test?
  3. A/B Test. Run your test and see what happens.

One of the key points was that we easily jump to testing things like the colours of buttons or email open rates (damn you MailChimp for making this so easy!) but before we just jump to the “easy” tests we need to collect better qualitative data which can guide our more quantitative A/B testing.

5 Sources for Qualitative Data

  1. Surveys. Always a great place to start. Make sure to keep them short and to the point, ensure the answers are actually useful and start open ended questions and progressively get more detailed and restricted in your answers.
  2. Feedback. You may have a formal “feedback” button or response form for your donors but absent of that get in touch with your “on the ground” fundraisers who are meeting with your donors regularly as they can provide some good insights to you on the donors behalf.
  3. Reach Out. If you have some phone numbers (from the survey for example) it can be very fruitful to follow up with a call. You can really dive deeper with the donor/respondent and get at the all important “why” behind their action or response.
  4. User Activity. With things like Google Analytics (and yes Kiss Metrics…) you can see some pretty interesting things that can be a good qualitative source for tests. People bouncing from a particular page? Getting lot’s of traffic from some social media channels but not others? What content to people share?
  5. Usability Tests. Using something like UserTesting.com you can watch and get real time data/feedback as users use your site. You can track their behaviour and see what stands out and then create a test to prove out the concept with your audience.

So where you should you start when A/B testing? With the qualitative insights to inform your predictions which can be tested A/B style. More to come in the coming weeks/months on A/B testing and if you are in Vancouver come check out the session on December 4th.