I’ve heard about UserTesting.com a few month back during an IxDA meeting and was intrigued. Then I ended up reading about it on UX Booth in a review they made of the service. At that time I was really getting interested in the service and decided to try it out as soon as the opportunity would present itself.

I finally got this opportunity back in December. Below is my a short summary of my experience with it.

Visit UserTesting.com

If I had to summarize my impressions in one word I’d choose: “Brilliant”. I was really amazed by the service offered by this company for a price that is near to nothing compared to the cost of usual research techniques.

What it is

UserTesting.com is a website that allows you to organize quick non-moderated user tests with users in the US, the UK and Australia. As a client, you basically register, enter the website you want to test, write up the tasks you want users to perform and fill in some quick user segmentation information (e.g. gender, age group, technological know-how). UserTesting will then publish your tasks to its pool of recruited users that they trained in think aloud technique. Users will in turn decide or not to take on the tasks.

The service costs $29 per user, which is a fraction of what a standard usability test costs and that includes the fee that users will receive for participating in the test.

The deliverable

What amazed me the most with this service was the speed of getting the results back. I published the test and in less than 4 hours had all 3 test videos back. Each video includes the screen recording of what the user does on the website with a synchronized audio of what they are saying (using think aloud) at that time. Video are supposed to be around 15 to 20 minutes depending on the tasks you have included in your test. In my case, videos were around 20 minutes, with one lasting 30 minutes as the user got sidetracked a bit and spend more time trying to complete the tasks.

The video deliverable is very much like a standard usability test deliverable. I found that users were pretty good at saying what they were thinking about as they were using the site. The deliverable also includes a short questionnaire, but this was quite useless in the end as the questions were too general (as of today, it is not possible to add your own questions to your study). In any case, with the speed of delivery, this opens the door to some potential extreme iterative process to improve your interface and test different design options.

What I got from it

I used the service to test a new walkthrough we implemented on GoAnimate for first time users. We included 3 users only in the study to test out the service. Basically I asked users to register on the site and head to our animation studio. Once in there our walkthrough would introduce them to the different basic feature of the studio to make sure they could start using the tool right away. I feel that this type of scenario was a great fit for using UserTesting.com. It was a straight forward linear process for users to follow that did not leave much space for users to get side-tracked. As a result, the resulting videos were really focused on the effectiveness of the interface in presenting the features of our tool.

Although all 3 users finally were able to complete all tasks, from the videos, we discovered some technical bugs that could prevent users from easily completing the walk-through. More importantly we discovered ways to improve the layout of the explanatory text and some interesting insights into how first time users might be confused by the structure and naming of things in our studio. It did not lead to any ground-breaking discovery, but did point out to really interesting issues and opened the path to more investigation into areas we had taken for granted. In this respect, the results reminded me of standard usability testing, where the results always end up revealing unexpected issues.

UserTesting vs. “Standard” Usability Testing

This said, I think I have a problem with this service taking up the name of usability testing. Personally, in almost all the usability tests I was involved in, I found that the discussion between the moderator and the user and especially the pre-test discussions always led to very interesting insights and opened new paths to improve a product. More than just testing an interface, usability tests are also an opportunity to learn more about the users and their goals. I would think that a moderated usability test that does not use this opportunity to probe more on users’ goals and motivations is missing a very important opportunity. Moderated usability testing offers great flexibility for a moderator to probe issues.

In any case, I find it very misleading to compare the price of a standard usability test and of  UserTesting’s service as they offer very different types and amounts of insights. In this sense I do not really feel UserTesting’s claim to be 1/10th of the price of competition is a valid claim. They are just not really comparable research tools. This does not change the fact though that UserTesting delivers excellent value for money.

Careful with the tasks

As I mentioned earlier, I think I used the service with a set of tasks that particularly suited the service well. It was a linear set of tasks that did not leave too many open doors for the testee to wander around. Keeping the users focused is a very important aspect of “standard” usability testing, but even more so with UserTesting.com. The user can very easily get side-tracked with an unclear or unfocused task and there will be no moderator to lead them back to the right path. This makes the service unsuitable for more holistic studies.

Conclusion

I would highly recommend UserTesting.com to anyone working on web interfaces and wanting to perform some kind of tests to gain more insights about the effectiveness of their design. It is a great and powerful tool in the hands of the right people. One needs to be very careful though that it is not a silver bullet that can replace other user research tools, and moderated usability testing especially. UserTesting.com is great to use in conjunction with other user research tools or possibly by itself for very specific design updates. Like any other research tools it can be abused, but that does not make it a lesser valuable tool anyway. A must try!