A/B testing or usability research—which one works the best? Both

Message me on LinkedIn if you’ve heard this before: your organization believes your website should provide more benefits to your bottom line than it currently does. But, you don’t know where to start in assessing which parts of the website could have the most impact on that bottom line. Each department has distinct opinions on where the site can perform better. Your users definitely know how the site could better serve them (primary research pro tip: they’ll tell you if you ask them). And Google Analytics tells another story about what could perform better for your KPIs and conversion goals. 

Consider the power of combining an ongoing A/B testing program and usability testing. Used in combination with each other, they can provide anywhere from a 10-50% lift in your website conversions. Let’s dive into a quick definition of each, then show how you can use each of these research tactics to have a real impact on website user conversions.


Let's define A/B testing

A/B testing splits your live website traffic into two or more parts. You determine what percentage of your audiences sees the A version, which leaves the remainder to see the B version with slightly different copy, page layout or calls to action. There are many tools to help you conduct a test like this—either through your content management system or through a tool such as Google Optimize. Allow the test to run for enough time to gain statistical validity. The key benefit to A/B testing is that you’re testing these variables on the live version of your website. So, real interactions from real users provide you with real data that you can share throughout your organization to show the impact of web changes on your users.


Let's define usability research

Usability research refers to a whole range of user-based research activities. This can include 1:1 interviews with representative users, user surveys, iterative design testing, market research and many other research methodologies. These types of research can require more engagement from staff and potential users because the questions you want answered tend to be deeper into the user experience of your website. The benefit to this type of research is the holistic knowledge you receive from allowing users to show you where the barriers are in the website experience.

Note before starting any user research or A/B test, you’ll want to have a hypothesis on what you believe to be true first. For instance, using a different image and H1 on a pivotal landing page could help you achieve a better bounce rate. Document that as your hypothesis for the test so you know which key performance indicator informs the success or failure of that A/B test. If bounce rates decrease significantly for B version of that landing page, you have a clear indicator of success.


Level of difficulty and cost

Each of the different methods of testing have varying levels of difficulty. A/B testing is very easy and cheap to assemble and publish to your website. You’ll need to develop your hypothesis, but that’s simply putting words down on paper and getting organizational buy-in to the idea. A/B testing is great at testing small tweaks and variables throughout your site. You can most likely create the assets needed for this type of testing in-house because the variables you test can be supported by your design and writing team.

The only other consideration is the cost of software to run a test. Google Optimize is free and your team is most likely familiar with the other Google Analytics suite of tools. I really like the comparison chart that Conversion Rate Experts put together on this topic and provides loads of software options and their relative benefits. Because A/B variate testing is relatively easy to conduct, we recommend running multiple A/B tests throughout the year. Your website lives in a very organic world and needs to stay optimized for conversion and the best possible user experience.

Usability research can scale from economical to expensive depending on the level of research you conduct and the number of participants you need to reach. A simple user survey is relatively easy to create, email out to your contacts and quickly get results that can show where current pain points are for the user. Talking one-on-one, conducting moderated research studies and assessing iterative designs can get costly due to the human labor involved in each of those types of testing. The learnings you’ll gain from the research almost always make the investment worth it. We live in an analytics world where you try to understand user behavior by looking at clicks and open rates. While that data is important, think about how much more valuable it is to get real human-based input on your website content. The investment of time and budget is worth it to gain insights beyond the clicks and understand what content and website structures actually help your users in their journey.


Research frequency matters

No matter which path you take in conducting research, there are very clear benefits to creating the ideal website experience possible for your users. Commit to making variable testing and research a perennial part of your marketing budget. A/B testing should be happening at all times to help with smaller website changes. Larger usability research projects can then be helpful to add to that knowledge once per year. Once you’ve identified that testing should play an essential role in your marketing program, all you really need to do is commit to investing in either of these types of research. Layout and design are low-hanging fruit to A/B test on immediately. Look at key landing pages on your website and find small ways that can be tested to make them better. Small tweaks to headlines, images and call to action placement can make a huge difference. In addition, you can conduct multiple tests at one time to really make your work more efficient. As you enter into larger usability tests, it does reduce the number of times you can realistically conduct experiments throughout the year but the benefits to that usability testing will also show larger leaps in insights.


Benefits to website testing and research

Each of the approaches covered in this article have an impact relative to their scale.


A/B testing normally shows you small, iterative improvements that can help you meet specific KPIs (bounce rate, time on page, click through rate, etc). Let’s say A/B testing leads you to a 5% improvement in a key KPI such as click through rate—that compounds across the year to a substantial improvement for a low cost investment like A/B testing.

On the other end of the spectrum, usability research can typically lead to game-changing insights that can literally have a 30-50% lift in your business outcomes and the end user experience. Simply put, you can make larger leaps in lift when you conduct usability research. For example, a research project on a potential market you’ve long considered entering might point the way toward opening up an entire new segment for your business or organization. The power of user and customer input coupled with your own organizational intelligence is real and cannot be duplicated by only thinking within our organizational walls.


A combination of both A/B testing and usability research will provide intelligence in your ongoing efforts to continually improve your website for your users and for your bottom line. So, lean in to learning by having your website users point the way forward and let us know if there’s a research project you can use our expertise on.