In short
Usability test refers to an empirical method for checking how well a website, an app or a piece of software can actually be operated. People from the target group complete realistic tasks while a test moderator or a recording observes where they hesitate, make mistakes or give up.
What matters is the observed behaviour, not the participants' opinion of the design.
This distinguishes the usability test from surveys, focus groups or internal reviews. A team knows its own product too well to recognise its hurdles reliably. Only when a person without prior knowledge tries to complete an order or submit a holiday request does it become visible whether navigation, terms and processes are understandable. A compact overview of the method is also provided by the Wikipedia article on usability testing.
What usability means according to ISO 9241-11
The yardstick for a usability test comes from the standard ISO 9241-11:2018 ("Ergonomics of human-system interaction – Part 11: Usability: Definitions and concepts"). The standard was published in its second edition in 2018 and, according to ISO, was reviewed and confirmed in 2023. It describes usability as the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.
This definition gives rise to the three dimensions that a usability test makes measurable:
- Effectiveness: do users achieve their goal completely and correctly? A typical metric is the success rate per task.
- Efficiency: what effort in time, clicks or mental work is needed for this? Measurements include processing time or the number of detours.
- Satisfaction: how do users experience the interaction? Standardised questionnaires such as the System Usability Scale are used for this.
The addition "in a specified context of use" is important. A warehouse app that works well in the office on a large monitor can fail on a scanner in the aisle when used with gloves. A serious usability test therefore reproduces the target group, tasks and environment as realistically as possible.
Variants: moderated, unmoderated, remote and on site
Usability tests can be distinguished along two axes: whether a moderator is present live, and whether the test takes place in the same room or over the internet. Both axes can be combined, so there are moderated tests on site as well as moderated remote tests.
Moderated usability test
In a moderated test, a test moderator accompanies the session in real time. They set the tasks, observe, ask follow-up questions and encourage participants to think aloud. According to an article by the Nielsen Norman Group on remote tests, the advantage lies in being able to ask questions directly. The difficulty is finding the right moment for a question without influencing behaviour.
Unmoderated usability test
In an unmoderated test, participants work alone and at a time of their choosing, usually via a testing platform that records the screen and voice. This saves appointments and allows many sessions in parallel. On the other hand, follow-up questions are only possible if formulated in advance, nobody helps if someone gets stuck, and thinking aloud is harder to enforce. Unmoderated tests are particularly suitable for narrowly defined questions, for example about a single form.
Remote or on site
Remote tests via video conference or testing platform reduce travel costs and make it easier to recruit target groups spread across the country. Tests on site, for example in a usability lab, at the customer's premises or directly at the workplace, on the other hand provide the real context of use: noise, devices, interruptions. For an app used in field service or production, a test at the place of use is often more meaningful than any simulation.
Methods in usability testing
A usability test is not a single tool but a framework in which several methods interact. The following three belong in almost every test.
Task scenarios
Instead of "just click around", participants are given concrete, realistic tasks, for example: "You need a day off next Friday. Request it." Good scenarios describe a goal, not the route to it, and avoid the terms used in the interface. If the menu says "Absences", the task should not use this word, otherwise you are only testing word search.
Thinking aloud
With thinking aloud, participants say out loud what they are looking for, expecting and understanding while they work. In his article "Thinking Aloud: The #1 Usability Tool" (2012), Jakob Nielsen describes the method as the most important usability tool: cheap, robust and usable in every project phase. As limits, he names the low statistical significance, the unfamiliar situation and the risk that clumsy follow-up questions distort behaviour.
System Usability Scale (SUS)
The System Usability Scale was developed in 1986 by John Brooke at Digital Equipment Corporation. It consists of ten statements rated on a five-point scale from "strongly disagree" to "strongly agree". For odd-numbered statements, 1 is subtracted from the value; for even-numbered statements, the value is subtracted from 5. The sum multiplied by 2.5 gives a score between 0 and 100.
According to MeasuringU, the average across 500 studies is 68. A score is explicitly not a percentage: 68 does not mean "68 per cent good" but average. The SUS is suitable for comparing versions of a product or products with each other. It does not reveal which specific place causes problems; that requires observation.
Depending on the question, eye tracking, click-path analyses or short interviews after each task are also used.
How many test participants do you need?
The best-known rule of thumb comes from Jakob Nielsen's article "Why You Only Need to Test with 5 Users" from 2000. Based on joint research with Tom Landauer, he describes how a qualitative test with five users uncovers around 85 per cent of usability problems. With each additional person, the proportion of new insights falls significantly, because the same problems recur.
The rule is often quoted in abbreviated form. Nielsen himself names several restrictions:
- Homogeneous target group: the calculation applies to comparable users. For two clearly different groups, such as dispatchers and drivers, he recommends three to four people per group; for three or more groups, three per group.
- Several small rounds: instead of 15 people in one test, he suggests three rounds of five each, with revisions in between. The second round shows whether the corrections work and uncovers deeper problems.
- Only for qualitative tests: anyone who wants to compare metrics such as success rate or processing time reliably needs considerably more participants.
For quantitative usability studies, the Nielsen Norman Group recommends 40 participants in most cases in a summary of sample sizes (Budiu and Moran, 2021). Around 30 are justifiable if you accept a confidence level of 90 instead of 95 per cent. 20 or fewer are possible but, in the authors' estimation, considerably riskier.
The process of a usability test in six steps
A structured process makes results comparable and prevents a test from becoming a non-committal demo. The following sequence has proven itself:
- Set goals: what should the test answer? For example: "Do new employees find the shift schedule without instruction?" Tasks and metrics are derived from the goals.
- Recruit participants: what is needed are real representatives of the target group, not colleagues from the project team. A short screener with questions on role, device and previous experience filters suitable people.
- Write scenarios and a guide: three to eight realistic tasks, a short introduction, follow-up questions after each task and, if applicable, the SUS questionnaire at the end.
- Run a pilot test: a trial run with one person shows whether tasks are misleading, too long or technically blocked.
- Conduct and record the tests: screen, voice and, if agreed, face are recorded. For recordings, you need the participants' informed consent.
- Evaluate and prioritise: observations are collected, condensed into problems and sorted by severity.
For the last step, Jakob Nielsen's severity scale has become established, described in the article "Severity Ratings for Usability Problems". It ranges from 0 (not a usability problem) via 1 (cosmetic), 2 (minor) and 3 (major) to 4 (usability catastrophe, to be fixed before release). The rating is based on frequency, impact and persistence, i.e. whether users overcome the problem once or keep stumbling over it.
When to test in the project?
A usability test does not need a finished product. Even paper sketches or a clickable prototype from a design tool are enough to clarify basic questions about navigation and terms. The earlier you test, the cheaper changes are, because no code has to be rebuilt yet. How wireframes and clickable prototypes fit into the design process is described in the article app design process: from wireframe to clickable prototype.
Later in the project, testing remains useful: before the launch of a minimum viable product (MVP), after major releases or when metrics such as abandonment rates point to a problem whose cause is unclear.
Distinction: usability test and A/B test
Usability tests and A/B tests are often confused but answer different questions. An A/B test distributes real traffic across two or more variants and measures which performs better on a predefined metric. According to the NN/g article "A/B Testing 101" (2024), it reliably shows what happens, but not why. In addition, it usually needs thousands of users for statistical significance.
The usability test works the other way round: few people, but deep insights into reasons and misunderstandings. In practice, the two complement each other. The usability test provides hypotheses and ideas for solutions, the A/B test checks their effect in live operation. For internal applications with a few hundred users, an A/B test is often not meaningfully feasible at all, whereas a usability test is.
Usability test and accessibility test
A classic usability test checks whether the target group can cope. It only checks whether people with disabilities can use the product if they are specifically among the participants. An accessibility test, on the other hand, assesses compliance with technical criteria, usually the WCAG (Web Content Accessibility Guidelines). For many digital services for consumers, accessibility has also been legally relevant in Germany since 28 June 2025 through the Accessibility Strengthening Act (BFSG); whether a service falls under it has to be checked on a case-by-case basis.
| Feature | Usability test | Accessibility test |
|---|---|---|
| Guiding question | Can the target group complete their tasks effectively, efficiently and satisfactorily? | Is the product perceivable, operable, understandable and robust for people with disabilities? |
| Yardstick | ISO 9241-11, task success, time, SUS | WCAG success criteria, referenced in the EU via EN 301 549 |
| Method | Observation of real users in task scenarios | Expert review, automated tools, tests with screen reader and keyboard |
| Participants | Representatives of the target group, qualitatively often five per round | Testers; ideally supplemented by users with disabilities |
| Result | Prioritised list of usability problems with severity | List of violations of specific criteria |
The two checks are not mutually exclusive. In its guidance "Involving Users in Evaluating Web Accessibility", the W3C's WAI initiative emphasises that a pure conformance check is not enough to determine whether a service is actually accessible. It recommends combining tests with users with disabilities with the WCAG check and not generalising the results of a few people.
Typical mistakes in usability tests
- The wrong participants: colleagues, project members or friends know the product or industry too well and skip exactly the hurdles that stop real users.
- Leading tasks: tasks that anticipate the click path or menu terms measure reading skills instead of usability.
- Helping instead of observing: anyone who intervenes at the first sign of uncertainty covers up the problem the test was supposed to find.
- Evaluating opinions instead of behaviour: statements such as "looks good" are not very reliable. What should count is whether the task was solved.
- Testing once and ticking it off: without a second round, it remains open whether the corrections work or create new problems.
Frequently asked questions about usability tests
What is the difference between usability and user experience?
According to ISO 9241-11, usability describes how effectively, efficiently and satisfactorily users achieve their goals. User experience is broader and also includes expectations, emotions and impressions before, during and after use. A usability test is therefore a central tool of UX work but does not cover all UX questions.
Are five test participants always enough?
No. Jakob Nielsen's rule of thumb applies to qualitative tests with a homogeneous target group, ideally in several rounds. With several user groups, you need three to four people per group; for quantitative studies, the Nielsen Norman Group usually recommends 40 participants.
How long does a single test session take?
A session typically lasts between 30 and 60 minutes. Longer sessions tire participants and distort the results. Plan fixed time slots for the introduction, tasks, follow-up questions and questionnaire, and check the total length in the pilot test.
Can you run a usability test with a prototype?
Yes. Clickable prototypes from design tools are well suited for checking navigation, terms and processes before development. There are limitations for aspects that require real data or real system behaviour, such as loading times, error messages or search results.
What does a usability test cost?
The costs depend on the number of participants, recruitment effort, format and depth of evaluation. An internal qualitative test with five employees is possible with manageable effort, while hard-to-reach target groups and studies on site are considerably more complex. Flat prices therefore cannot be given seriously.