You are here

Cool module: A/B Testing

Use Case

Usability testing can be so useful and telling and surprising to us when we're rolling out new pages, or designs/layouts or content, but it can also be so intensive to conduct.  From lack of expertise in running tests properly (if doing it within the organization) to lack of funds (if outsourcing the work), there are a myriad issues that can prevent us from doing good usability testing and simply rolling out changes instantaneously.

Enter one module I ran into recently, which is a very light-weight solution to testing and comparing different content, designs, etc.  The Multivariate module can help us run A/B testing on our website, and automates all of the following:

  • Presents site visitors with one of the element variants we are testing
  • Tracks whether the user has interacted with this element variant in the way we consider successful (visited a page, submitted a form, etc)
  • Creates a report presenting us with how our alternate element variants compare, based on the percentage of times each resulted in a successful event


Let's set up a sample test to see how this module can help us compare anything from alternate article headings, menu styles, block styles, etc.

Module Setup

  1. There are a lot of sub-modules with this module, and all will appear in the 'Usability Testing Suite' section.  Enable the ones that correspond to the variant types and success conditions you are interested in.  For this example, we'll need to enable at least the following
    • Multivariate - base module required by everything else
    • Internal CSS mutation - With this module enabled, we'll create our second variant by adding css manually.  (Our first variant will just be the theme's default styling)
    • Path success condition - This module will allow us to define the visiting of a certain page (clicking a link) as 'success' for a certain variant
  2. Go to /multivariate to create your first A/B test
  3. Let's name our first test "Menu style test"
  4. Make sure the 'Enabled' checkbox is checked
  5. In the 'Run conditions' tab, decide whether this is testing element variants on a certain page (in which case, you can use the 'Front page' or 'Path' conditions), or - if you're testing a main menu variation, which appears on every page, just choose 'Current theme' as the condition, and select the non-admin theme that appears for anonymous users.
  6. In the 'Sets and variants' tab is where all the magic happens
    1. Let's create a set called 'Main menu style' and give it two variants: 'Light style' and 'Dark style'
    2. At the bottom, select that we'll create the variants through 'Internal CSS'.  As soon as you do this, the Internal CSS boxes will show up under both variants, as you see in my screenshot
    3. I have not applied any special CSS to the 'Light style' variant, because that's the variant that my custom theme has already styled
    4. I have applied css to the second variant (just making it a different, darker color with white text, for example's sake).  This will override the theme's CSS
  7. Finally, in the 'Success conditions' tab, I have enabled the 'Path' condition.
    1. Think about what constitutes 'success' for the element you're testing. 
    2. In my case, I'm trying to see which style (color) menu entices more people to interact with it, so success, for either variant, is someone clicking on a menu item. 
    3. I have added four paths, corresponding to a click on any of the four menu items (and displaying a variety of ways you can input the paths)


Running the A/B Testing

As soon as you save this test (and if you've enabled it by checking the 'Enabled' checkbox), it'll be live and running!

When visitors go to your website (or the certain page you're running the test for under 'Run conditions'), this module will ensure that *one* of the variants you're testing against each other is presented to the user.  How cool!  Then, if the user completes an action set in the 'Success conditions' tab, that'll be logged as a success for that particular variant.

I went to my site a couple of times, and, sure enough, I was presented with one of my variants at random:

Variant 1:

Variant 2:


I clicked on the menu items sometimes, navigated to places via blocks or other links or just refreshed the page other times.  Now - let's see the results of our test.

Test Results

To see how your test is doing, simply go to /multivariate on your site and click on the test name.  You'll see a table like the following:

It'll show you, for each variant that you're testing, how many times it was displayed to a user (45 each, in my case), and how many times the user completed the action you were aiming for (the 'Success condition').  As you can see, the 'Light style' variant did better in my case.  I did this on purpose, since I like it better and wanted it to win.

What now?

When you feel you have a sample size of responses that is big enough, simply disable the test, and roll-out the variant that was more successful. :)




Add new comment

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.