moderated vs unmoderated usability A/B testing FI

Posted on March 29, 2022

Usability A/B Testing: Moderated vs. Unmoderated

By Owen Fay . Posted on March 29, 2022

In the 40-plus years usability testing has been around we’ve seen improvements and changes that can make it hard to choose what kind of test to do. Tests used to only be done in a lab, with highly recruited participants, costing a lot of time and money.

Today, in less than an hour, you can test with people in different countries for a few bucks, all while sitting at your home or office on your computer. The most popular methods are now a moderated test and an unmoderated test.

Both have their advantages and there have been some big changes with each that make it easier for you to get great feedback. In this blog, we’ll show you what each usability A/B testing process is and our opinion of what might be better.

Moderated Usability A/B Testing

Here’s how I would sum up moderated usability testing: You ask someone to use a website, app, microwave, whatever, and listen to them while you watch them use it. It’s a more traditional way of testing but can still be done online with some added resources.

It’s what most people think of as usability testing, mostly done in a lab or just in person, but modern tech lets you do this remotely, reducing some of the high cost and time commitment. The business will ask the testers to do some tasks, guide, and watch them as they go through the whole test.

When you decide to do moderated testing the moderator should have some experience running a test. In our experience, moderated tests can be expensive and time-consuming but will also give you great feedback if you do it the right way.

When To Do A Moderated Test

Moderated tests are great when you want to interact with your users and see how they use your website or business. The moderator will get great info on how the testers feel and what their experiences really are. You can take all of the results and make great changes to your business because you watch how people are going to experience your resources.

I suggest doing moderated usability A/B testing when you come up with a new feature, product, or way of doing things because you get gut reactions from the testers.

Unmoderated Usability A/B Testing

Here’s how I would sum up unmoderated usability A/B testing: You ask someone to use a website, app, microwave, whatever, and listen to them while you watch them use it.

No that’s not a typo, yes I did explain moderated and unmoderated tests the same way. That’s because they’re essentially the same thing with one difference, the way the users complete the test.

In moderated testing, a moderator guides the users and is present in a test, with unmoderated testing the testers do everything on their own without someone watching them complete tasks or answer questions.

Unmoderated usability A/B testing is a lot different than traditional testing and only became doable in the last 10 or so years. So not everyone knows about it and how it might even be a better way to test your business.

The entire process can be more user and business-friendly, the whole process including paying participants and setting up a test is taken over by the platform. The business has to invest less in the whole process of usability A/B testing.

Now all you need to do is enter the URL of the website or page you want to test, choose how many people you want responses from, and get feedback from all of them individually.

When To Do An Unmoderated Test

Most of the time unmoderated testing is good for getting quantitative data, but new tools like Poll the People let you get written feedback too. Just like moderated testing, it can be done at any stage of the process. But because it’s often quicker, less expensive, and easier it might be more practical to run unmoderated tests at any stage of your development.

Moderated vs. Unmoderated Usability A/B Testing

As we said, the real meaning of both kinds of tests is the same with the only difference being if you are guiding and watching testers or letting them do everything on their own. I’ll dive a little deeper into the pros, cons, and a comparison of each method to paint the full picture.

Pros Of Moderated Testing

Moderated usability A/B testing gives you more control. Being there with the participants (online or in-person) lets you give them support and guidance. You will be able to see what they are doing and help if there are issues. You’ll also be able to ask clarifying questions and clear up any confusion that the moderator might have.
Moderated testing can create more engaged participants. Having the ability to have a natural conversation with the people taking the test can create trust and make them more likely to be engaged with the test. The moderator can get a good sense of how everything is going, it might even give you feedback that the testers wouldn’t have just written down.

Cons Of Moderated Testing

Moderated testing is usually more expensive and time-consuming. You usually have to personally recruit the tester and because you need someone watching the test it can take a lot more time to put together. You need to plan all of these things and ask individual questions watching multiple people, that takes time and effort. Plus, a lot of the platforms that let you do moderate testing are more expensive than unmoderated.
You need more technology and resources. You need a moderator with some experience in usability testing and some technology that lets you record, watch, and analyze the participants. A good unmoderated platform has everything you need already set up.
Guiding and watching the users can create bias. The last thing anyone wants is to spend time and money on a test and get meaningless results. Sometimes when someone is guiding the test they can steer the test in the direction they want it to go, destroying the value of the feedback.

Pros Of Unmoderated Testing

Unmoderated usability A/B testing is quicker and less expensive. Let’s just look at Poll the People, the average test takes 30-60 minutes to finish. A lot of moderated tests (and other unmoderated tests) take days and sometimes weeks to complete. Unmoderated testing, 9 times out of 10 is far less expensive, you don’t need a lab, a dedicated moderator, or as many complicated resources. Looking at Poll the People again, we have all the technology you need and a test with 200 users costs less than $100.
There’s less chance for bias by letting users do everything on their own. You still need to give some directions and ask the right questions but the testers won’t be guided one way or another. Participants interact with the business naturally and exactly how they would if they just found you with a google search.
You enjoy more flexibility with the testing process. There’s no need to have a dedicated moderator or set aside time to do the test. You can plan, create, and run a test quickly and let the participants moderate themselves. Testing can be done at any time, anywhere, with just about anyone.

Cons Of Unmoderated Testing

You can’t ask follow-up questions or give support. When no one is there watching participants you don’t know if they have issues, you can give them help if they are confused but their unguided feedback might help you realize what you’re testing isn’t ready. You also can’t ask clarifying questions if you are confused, unfortunately, it’s impossible to ask follow-up questions or clarify some of the feedback.

Our Recommendation

In our opinion usability, A/B testing is about one thing, actually doing it and doing it consistently. So if you have the time, money, resources, and think moderated testing is good for you, go for it. Testing to optimize your business is the most important thing.

But, I’d say 95% of businesses don’t have the money, time, or experience to follow a good moderated testing process or they might not want to dedicate that much to the process. For us having the flexibility of unmoderated testing is much more appealing.

You might see other sources saying you can’t get the most reliable data from unmoderated tests, I’m here to tell you they’re wrong. Usability A/B testing has evolved a lot in the last few years and you will see the same benefits as a moderated test while spending less money, time, and effort.

NN Group Study

You might think I’m being biased so I’ll give you some facts. Nngoup.com did a study on the cost of each test, the low estimate for a moderated test with 20-40 participants is about $450 with 32 hours of work, and the high estimate is about $1,600 with 48 hours of work. They say a low estimate for the same unmoderated test is about $250 with 11 hours of work and the high estimate is about $1,200 with 27 hours of work.

You can see the difference clearly, but if we take a look at Poll the People the difference is even bigger. If we use the same criteria a test with 20-40 participants costs less than $50. Nngroup says it takes about 4 hours of planning, and with that audience size, a test will complete in about 30 minutes on our site. Adding about 2 hours for analysis, a test with Poll the People, planning, running, analyzing takes just over 6 hours.

If we just focus on the time and money it takes to run the test it takes over two hours less to test on our platform than other unmoderated tools and costs $200 less than the low estimate.

I’m not sure about you but the benefits of testing with Poll the People are pretty clear. We offer great templates for testing, have all of the resources you might need to create valuable tests and do it easier and cheaper than competitors.

If you want to learn more about our platform or just how usability A/B testing can be used, some tips, mistakes, or anything else check out our blog and usability A/B testing page.

We offer a free version or subscriptions with added features and benefits, we have pricing for every business at every stage. Once you are ready to sign up for our platform, start optimizing your business now and get the most out of your customers.

Author
Recent Posts

Owen Fay

Owen is a digital marketer at Poll the People an emerging user research and usability testing platform. Owen focuses on content creation and moving Poll the People forward through all digital marketing mediums. He is a graduate from Bryant University with a concentration in marketing, living in Boston.

Usability A/B Testing: Moderated vs. Unmoderated