A Comprehensive Review and Matrix of 14 Usability Testing Tools
In 2009, I published a well-received article on 24 usability testing tools. Since then, there has been significant growth in both the number and functionality of usability testing tools. An update to the article is necessary, but due to the wide range of tools now available, a clearer definition and a matrix of usability tools are essential. Therefore, the following matrix and detailed review of 14 usability testing tools will assist practitioners seeking tools for specific testing needs.
Usability Testing Tools Matrix
For purposes of classifying tools to evaluate in this matrix, the tools must have three critical functions necessary to conduct a classically defined usability test:
- Audio and video recordings of testers (ala Think Out Loud method)
- Ability to edit recordings of tester sessions into highlight reels
- Capability to test using mobile devices (Smartphones and tablets)
The classical definition of usability testing is a systematic observation under controlled conditions to evaluate how well testers can complete critical tasks.
Having the ability to record testers, observe their interactions, and hear their thoughts (similar to the Think Out Loud method) is essential to the observation process. It is also important to edit those recordings so that key insights can be shared in highlight reels. Furthermore, the growing use of mobile devices to interact with websites and apps makes it necessary to conduct usability tests for the mobile user experience. Therefore, tools should be capable of testing both mobile and PC experiences.
Not included in this review are Feedback or Click Tracking tools. This is because they only provide feedback or opinions about user interfaces, and lack the ability to observe testers conducting tasks. I will conduct the same comprehensive review of Feedback and Click Tracking tools in a future article.
Four Quadrants of the Usability Testing Tools Matrix
Usability testing tools are divided into four quadrants on the matrix, based on where they fall when evaluated against two critical attributes:
- Depth of Technology: What functionality and features are available with the tool, and how deep does the array of features run? Tools that score higher going up the vertical Depth of Technology axis have more functions and features.
- Tool Impact: This is a measure of the ability of the tool to impact overall testing, and thus the observations and eventual recommendations for usability improvements. Tools that score higher (going to the right) on the Tool Impact axis have the ability to have a greater impact on overall testing results.
Based on these definitions, tools can fall into one of four quadrants on the matrix: Authority, Contender, Innovator and Niche.
- Authority tools are those that provide a greater amount of usability testing functionality as well as a higher potential impact on discovering, diagnosing and ultimately optimizing the user experience.
- Contender tools have the ability to provide a greater impact in terms of their depth of technology, but may have more limited potential impact in discovering, diagnosing and optimizing the user experience
- Innovator tools have a more limited depth of technology, but have the potential to have a greater impact in usability testing and recommendations
- Niche tools have a relatively limited depth of technology as well as a lower potential impact on usability testing and recommendations results, but can still be very useful for the specific tasks they were created for
All of these tools have specific features that make them unique. Depending on your testing needs, any of these tools may be good choices and can compliment your toolkit.
Mobile Usability Testing Still Rudimentary
The upper right corner (Authority) of the matrix is blank for a very good reason. It represents the current lack of a fully integrated mobile usability testing solution. In my opinion we still lack a comprehensive, fully integrated mobile device usability testing tool, something that records both mobile device screen interaction and the audio/video of testers as they try to accomplish tasks. Because they are mobile devices, testing in this manner should be possible anywhere the tester goes, at any time, and using any smartphone or tablet (as long as it has a camera). Real time streaming of the screen and audio/video of the tester should be available for remote teams.
Mobile usability testing is currently in a neophyte stage. Yes, there are cludgy and cumbersome ways to test usability with a mobile device, typically requiring testers to place their phone on a desk with a camera hovering over it (thus not being at all ‘mobile’). Most methods today lack the sophistication of being a truly integrated and mobile experience.
Consider how just a few years ago new tools like Camtasia, Morae, UserTesting and others made usability testing websites an anytime, anywhere, quick and cost effective solution versus using expensive and cludgy labs or home grown equipment. We need that same fully integrated experience for mobile, but we do not have it, yet.
Authority Usability Testing Tools:
Ovo Logger by Ovo Studios can be considered to be the Rolls Royce of usability testing solutions. This is the tool for large Fortune 1000 enterprises or usability testing studios that conduct a high volume of testing and have teams spread around the globe. Typically these situations call for the need to have a robust tool to capture, edit and share results. It should be noted that Ovo Studios creates custom usability testing labs, and the Ovo Logger fits in well with those labs. The tool scores at the top of comparable tools for depth of technology, but is slightly limited in terms of impact due to requiring a dedicated PC, and being somewhat expensive to deploy.
Pros: Enterprise level usability testing software and hardware that records multiple inputs from PCs or Macs (cameras, microphones, screen interaction) and streams the output to observers. Editing tools enable real-time analysis of tests with the ability of observers to document important points. Can capture un-tethered iPads, non-jail broken iPhones and Android devices if using the Logger mobile app.
Cons: Designed specifically for larger firms with multiple locations and or usability teams. Requires a dedicated PC to run the software. Pricing is more expensive versus other usability testing tools and is based on number of sources, associated lab requirements, etc.
Tester Audio and Video Recordings? Yes
Recordings can be edited? Yes
Mobile testing capable? Yes
Price: $3,000 and up depending on scope of associated usability lab
Camtasia plus a Screen Sharing Tool
A very powerful Do It Yourself approach for remote usability testing is to use a screen capture tool such as Camtasia (from TechSmith) coupled with a screen sharing tool such as GoToMeeting, WebEx or similar applications. The major benefit of this approach is the ability to conduct moderated remote tests, because the practitioner has a direct connection to the participant. In addition, the recording of the screen interaction with the voice and or video of the tester is immediately available for editing on the practitioners computer. The downside to this approach is lack of a simple mobile solution.
Pros: Enables remote moderated usability testing, because the practitioner has a direct connection to the participant. Sessions can be shared with other remote observers using the screen sharing tool. Audio and video recordings of the screen interaction with the voice of the tester are immediately available for editing on the practitioners computer.
Cons: For part-time or ad-hoc uses, this approach may be more expensive than using some of the other remote unmoderated tools available from vendors. Screen sharing tools are notorious for occasionally having technical problems that cause poor connection performance, or worst case dropping the connection entirely. This is not an optimal solution for conducting mobile usability testing, although cludgy methods of using a webcam pointed at the testers mobile device can be tried.
Tester Audio and Video Recordings? Yes
Recordings can be edited? Yes
Mobile testing capable? No
Price: $299 for Camtasia, $19/month for GoToMeeting or $24/month for Webex
UserTesting provides a network of pre-screened testers who can meet several specified demographics. Or, practitioners can invite their own testers using the tool. Practitioners create a set of tasks for the testers to complete, either on your website or even competitor sites. Testers screen interaction and voices are recorded. UserTesting includes powerful and easy to use video editing tools to create highlight reels. UserTesting also provides mobile usability testing, which is a major feature considering the increasing utilization of mobile devices. Written responses to questions are included. UserTesting is one of the most robust tools to use for remote unmoderated usability testing and thus scores higher for depth of technology and testing impact on the matrix.
Pros: Relatively low cost way to conduct remote unmoderated usability testing. Audio and video of the tester are recorded, and editing tools enable highlight reels to be created easily and quickly. Mobile device usability testing is available. The large network of testers typically means testing session results are available in an hour or two. Demographic filtering tools enable finding the appropriate testers for most Persona situations.
Cons: Because UserTesting pays small fees to the testers for each completed test, it is critical to carefully create non-biased tasks, else you risk the tester trying to cut corners to accomplish the test as quickly as possible. As a practitioner, you must watch for this kind of activity and request a re-do of that test if you are concerned about the quality of the tester. As an alternative, using your own testers can alleviate much of this. Mobile testing is available, but requires the tester to be at a desk so a webcam can be pointed down to the screen, which is not typically how most people use their mobile devices.
Tester Audio and Video Recordings? Yes
Recordings can be edited? Yes
Mobile testing capable? Yes
Price: $49 per test with bulk pricing discounts available
Morae from Techsmith can be considered the Granddaddy of usability testing tools. This is because it was the first tool specifically created for usability testing. Even today it is still one of the best. It provides a comprehensive set of features available to record, analyze, edit and share usability testing sessions. Morae is an application that resides on the practitioners computer (PC only, sorry Macs). Morae records all screen interaction, including the testers voice and or face if using a webcam. Testers can be on any Mac or PC if testing websites or applications. Morae includes very powerful documentation, editing and sharing tools. Morae scores fairly high for depth of technology, and also fairly high for testing impact, the limiters being lack of a Mac practitioner solution and the issue of cludgy mobile testing.
Pros: Powerful screen capture and editing tools make this the choice for larger scale usability testing. Morae includes additional tools that can track a variety of screen and click interactions, plus has the ability to incorporate survey/opinion based data. Sharing test sessions with other practitioners and testing results with teams is relatively easy and quick.
Cons: Desktop & Laptop based system with no ability to record Mobile device screen interaction directly. Relatively expensive and PC-only (not Mac compatible) for the Manager and Recorder tools.
Tester Audio and Video Recordings? Yes
Recordings can be edited? Yes
Mobile testing capable? No
Price: $1,995
Contender Usability Testing Tools:
Userlytics provides both PC and Mobile based remote unmoderated usability testing. For mobile testing, the more expensive Gold or Enterprise plans are required. Testers can come from the Userlytics panel, or from testers you provide. Screen interaction, tester voices and webcam video images of testers can be recorded. Written answers to post test questions are also included. The depth of technology is good, but because of the requirement to use the more expensive Gold plans for downloading and editing of videos, capturing of sessions longer than 15 minutes and testing mobile, this tools has somewhat limited testing impact versus some competitors. Practitioners that conduct larger volumes of tests should consider Userlytics.
Pros: Userlytics is relatively low in cost, and is definitely easy to set up and use. The more expensive plans allow download of recordings for editing and sharing. In addition, images, videos and other assets can be uploaded to the Userlytics secure servers for early prototype testing. White label testing results sharing is possible for Gold and above plans.
Cons: Userlytics has the same issues as any other remote unmoderated usability testing tool, with the burden being on the test creator to ensure non-biased questions. Userlytics has a rather confusing set of test functionality based on plans (Basic, Silver, Gold, Enterprise). For example, tests recordings are less than 15 minutes for sub-Gold plans, and mobile testing is only available on Gold and above plans. Depending on the plan (Silver or above), each monthly subscription price includes one or more ‘free’ tests, with each additional test costing $33 or less per test. There are also limits in terms of number of post test questions that can be asked based on plans.
Tester Audio and Video Recordings? Yes
Recordings can be edited? Yes
Mobile testing capable? Yes
Price: $39 per month or more with bulk discount pricing available
UserFeel.com provides an assortment of testers, many of whom speak languages other than English, for remote unmoderated usability testing. Having the ability to choose a language other than English (many are offered) is a great benefit for testers of websites that are targeted for other countries. Written responses to post test questions are also available. Videos are available of the test on the website (but not available for download for editing purposes), and a white label playback option of the video is free. The depth of technology is good, but due to the lack of mobile testing and inability to download test recordings the tool impact is lower than some competitors. For those conducting usability testing in languages other than English, this could be a good solution.
Pros: A good tool for non-English language remote unmoderated usability testing. Demographic and language filtering tools are available. Pricing is relatively low and post test questionnaire functionality is available. The free white label playback of the test recordings is a plus.
Cons: Like other remote unmoderated usability tools, practitioners must be careful to create non-biased tasks or else risk having testers try to quickly finish the test, causing poor test results. The advanced video editing capabilities of some competitor tools are not available. Videos cannot be downloaded and edited. Mobile testing is not available.
Tester Audio and Video Recordings? Yes
Recordings can be edited? No
Mobile testing capable? No
Price: $39 per month with bulk discount pricing available
Innovator Usability Testing Tools
OpenHallway is a remote unmoderated usability testing tool for PC or Mac-based research. Testers come from the practitioner (no panel is offered) via a link that is generated by OpenHallway when the scenario and tasks are set up. OpenHallway requires no code to use, which makes it a great tool for testing applications, websites and even competitor websites. Screen interaction is recorded, along with voice and or the testers face, if the tester has a webcam. Plans (Basic, Plus and Premium) determine the amount of video storage available. Unlimited number of test sessions are available, however the video storage amount will act as the limiter to how many sessions are actually conducted. Because of video storage limits, and videos being MP4 only, and no mobile solution, OpenHallway has somewhat more limited depth of technology than some competitors, however the tool impact is good in that it can be used on almost any website with no code required.
Pros: OpenHallway is simple to use, easy to set up and therefore is a handy tool for recording unmoderated remote usability tests. As of the writing of this article there is a free trial.
Cons: OpenHallway is PC or Mac compatible, but does not support mobile. Although no code is required, Java must be present on the tester computer for the tool to work. Videos are MP4 only, and only the Premium plan ($199 per month) allows downloading (and thus editing) of the video.
Tester Audio and Video Recordings? Yes
Recordings can be edited? Yes
Mobile testing capable? No
Price: $49 per month or higher, depending on the plan
Usability provides a variety of usability, feedback and click tracking tools, with the primary advantage being they all use a single practitioner interface. This makes creating tests that require multiple tools much easier. There are two versions, a UX Suite and Conversion Suite. The UX Suite contains click testing, web testing, survey, card sorting and a Persona Creator. Practitioners can set up web tests that provide scenarios to website visitors, who then try to accomplish the tasks and click success or abandon buttons based on whether they thought they could accomplish the task. Results are provided in data and charts. No code is required unless you choose to utilize their website intercept request on your site. The lack of tester audio and video recordings for PC and mobile is a limiter for depth of technology. However, having the ability to utilize the tools efficiently and quickly provides a good tool impact.
Pros: A toolkit of the most common tools needed to fully understand the user experience. Having all tools in a single practitioner interface streamlines the process of creating and running multiple tool tests. In larger quantity testing scenarios, this set of tools can be more economical than using separate usability tools.
Cons: UsabilityTools does not provide audio and or video recordings of tester sessions. Mobile interaction using audio and video recordings is also not available. Pricing is not available on their website, although as of Sept. 2013 they had one time fees ($300) and monthly fees starting at $69 for the Professional version. Free panels of testers are not offered, although paid sources are available.
Tester Audio and Video Recordings? No
Recordings can be edited? No
Mobile testing capable? No
Price: Pricing information is not available on their website.
Loop11 provides the ability to conduct unmoderated remote usability testing on PCs, Macs or mobile devices. The tool is easy to set up and fairly easy to administer (no code required). A key advantage is it can provide real-time data. Loop11 provides useful information for evaluating task flow on websites, including competitor websites. No audio and video recordings of unique test sessions are available, and because of this the depth of technology, and tool impact are lower than some competitors.
Pros: Loop11 requires no code, thus is useful for conducting comparison tests of user experience on competitor or other websites. Setting up and running tests is relatively quick and easy. Real time information can be advantageous for situations where quick results are needed.
Cons: As with other unmoderated tools Loop11 does not allow interaction with the testers to probe or ask follow-up questions. The test creator has to find his or her own test participants. No ability to record the tester via ‘think out loud’ methodology, something that exists with competitor tools. Somewhat expensive versus competitors.
Tester Audio and Video Recordings? No
Recordings can be edited? No
Mobile testing capable? Yes
Price: $350 per project or $158 per month
UserZoom includes multiple tools including remote unmoderated testing on PCs and mobile devices (via UZ Mobile), along with other tools such as card sorts, tree testing, screenshot click testing and more. For the remote unmoderated usability testing tool UserZoom enables tasks to be created and deployed to either 3rd party panels or the practitioner’s own panel. Intercepts are also available, a handy feature when trying to reach actual visitors to a website. UserZoom does include video recordings of screen interaction, but only at the pricey $9,000 per year or higher plans, and no voice capture of testers are available, thus Think Out Loud recordings of users explaining their actions are not captured. This limits the depth of technology and tool impact when compared to some competitors.
Pros: UserZoom includes powerful reporting and analytics tools that can greatly help researchers synthesize and analyze post test data, including heatmaps, time on task reports and much more. Plans allow practitioners to pick the best price point and functionality for their needs. Plans include a free Pilot, UZ Proof of Concept at $1,000, Basic plan at $9,000, Pro plan at $29,000 and an Enterprise plan that requires a custom quote. As of the writing of this article UserZoom offers a free pilot study.
Cons: UserZoom does not enable voice and or video capture of the tester as they try to accomplish a task, a major differentiator compared to competitor services. UserZoom does not have its own panel of testers available for free, as other competitors do. Practitioners can view and edit recordings of testers while they were conducting tasks to make highlight reels, but there is no audio nor video capture of the actual tester. UserZoom is rather expensive versus its competitors, and lacking the ability to hear and see testers via session recordings while they were conducting the task is another major differentiator between UserZoom and other remote unmoderated usability testing tools.
Tester Audio and Video Recordings? No
Recordings can be edited? Yes, but only recordings of screen interaction (no tester voice is available)
Mobile testing capable? Yes
Price: $9,000 for the Basic plan and up
Niche Usability Testing Tools
TrymyUI.com is another of the many SaaS-based remote unmoderated usability testing tools. Testers create tasks, and videos are provided that include screen interaction and the testers voice. Written answers to post questions are also included. As of the writing of this article a free trial is included. Practitioners can use their own testers, but at a license cost of $99 for a single use test. Because video editing and mobile testing are not provided, the depth of technology and testing impact are lower than some competitors.
Pros: TryMyUI provides an alternative to other remote unmoderated tools that is slightly cheaper than many competitors. As a tool that provides video and audio recording of testers as they try to complete tasks it fits the classical definition of a usability testing tool.
Cons: All the same issues with remote unmoderated usability testing apply, with the burden being on the practitioner to create non-biased questions, and to carefully analyze the results to ensure the tester was not trying to use short cuts to complete the test. Video editing is not available, nor is mobile testing.
Tester Audio and Video Recordings? Yes
Recordings can be edited? No
Mobile testing capable? No
Price: $35 per test with bulk discount pricing available
WebEffective by Keynote Systems provides remote, unmoderated usability testing via tasks that are assigned and recorded. A panel is available, or practitioners can use their own testers. Data including click stream information is captured and displayed in a variety of reports, which can be downloaded. Individual sessions showing screen interaction are captured, however there is no audio nor video commentary captured of the tester as they go through the tasks. Mobile testing is available. The lack of audio and video of testers as they try to complete tasks means WebEffective has a lower depth of technology and testing impact than some of its competitors.
Pros: A powerful tool to capture user interaction on websites or mobile devices, with a wide variety of reports available.
Cons: WebEffective does not provide audio and or video recordings of tester sessions. In this regard, WebEffective operates more as a click stream tool than a true usability testing tool.
Tester Audio and Video Recordings? No
Recordings can be edited? No
Mobile testing capable? Yes
Price: Pricing is not available on the Keynote WebEffective website
The Ovo Solo tool by Ovo Studios provides a small business version of Ovo Logger that enables screen interaction video capture along with the testers face and voice. Simple logging controls enable analysis of the resulting test videos. However the tool is PC only. The relatively low price point makes this a good candidate for practitioners on a tight budget. Because of the PC only nature of the tool, the basic editing functions and the inability to record mobile sessions, this tool has lower depth of technology and testing impact than some of its competitors.
Pros: The relatively low price point makes this a very affordable way to capture, edit and display usability testing sessions.
Cons: This tool is PC only and is not able to record screen interaction on mobile devices. Editing tools are basic and may not meet more advanced usability tester needs.
Tester Audio and Video Recordings? Yes (but tool is PC only)
Recordings can be edited? Yes
Mobile testing capable? No
Price: $99
For Mac users, Silverback from Clearleft enables remote and in-person moderated usability testing. Silverback is not available for PC based practitioners. Session screen interaction and audio and or video of the tester are recorded, and simple editing features like task tagging are available. Recordings are saved in QuickTime, which makes editing of highlight videos relatively easy for Mac users. The lack of PC functionality and video editing capability means this tool has lower depth of technology and testing impact than some other competitors. Mac practitioners will find Silverback a very easy to use tool at a very affordable price.
Pros: A relatively low cost and easy to use tool for Mac users that provides the basics in terms of classical usability testing functionality.
Cons: Mac desktop and laptop only, does not work with PCs. No ability to record mobile device screen interaction directly. As of the writing of this article currently does not support FaceTime camera with newer Macbooks, but the app is being re-written to fix this. Lacks powerful editing tools available in other applications.
Tester Audio and Video Recordings? Yes (but tool is Mac only)
Recordings can be edited? No
Mobile testing capable? Yes
Price: $69.95
14 Usability Testing Tools Conclusion
Any of the 14 usability testing tools reviewed here can provide excellent results if applied in a manner which best fits their unique functionality and features.
If Your Favorite Tool is Missing
Not seeing your tool here? It could be because I’m classifying it as a Feedback or Click Tracking analytics type of tool. My original 24 usability testing tools article included many of these tools, but also included some tools that can better be defined as Feedback or Click Tracking analytics tools. In future articles I will provide a matrix and reviews separately for those types of tools. And if your tool is not a feedback or click tracking tool, let me know by leaving a comment.