How to measure usability? (Part II: Usability metrics)

In the first part of “How to measure usability?", we explored the purpose and usefulness of metrics. In this article, we give advice on what to keep in mind upon picking metrics and how to start creating a system. The third part gives an overview of how to implement the metrics system and analyse the results.

WHERE TO BEGIN?

Before starting to use metrics, a few things should be considered to ensure that the metrics are as useful as possible and that the end result is satisfactory.

What should be considered before creating a metrics system?

1. THINK ABOUT WHAT YOU WANT TO MEASURE AND WHY

First of all, you should think of the specific goals that you wish to achieve by using metrics. One of the ways is to set primary and secondary goals and try to achieve them in separate stages.

A few examples of possible goals:

-  analysing system performance

-  detecting parts of the user interface that cause issues

-  analysing and improving user satisfaction

-  comparing different versions and updates to evaluate system usability

By setting specific goals, it is also easier to work out suitable categories and metrics for each category.

2. CONSIDER THE USEFULNESS AND NECESSITY OF DIFFERENT CATEGORIES

By taking previously set goals into account, you are now able to determine which categories are more important upon collecting necessary information. Perhaps one of the traditional categories does not meet your needs at all and it would be more practical to use an alternative option.

We recommend having at least 4–5 categories as this ensures a more comprehensive result for the usability score.

3. INVOLVE EMPLOYEES WITH TECHNICAL KNOWLEDGE OF YOUR INFORMATION SYSTEM IN CREATING METRICS

People, who are already familiar with the system and are involved with maintaining and improving it daily, are able to suggest the right categories and metrics for this particular system. Developers, designers, system analysts or marketers informed about web statistics could offer you a lot of help. Brainstorm together for ideas or involve them in the whole process!

4. KNOW THE USERS AND THEIR BEHAVIOUR

Do not fixate only on system performance (e.g., measuring how quickly the system is able to load a page), but also analyse user behaviour. User behaviour could be negative (halting the process, clicking on wrong elements, getting lost in the system) or positive (completing the process, clicking the right buttons, using the system faster with every use) and both should be taken into account upon creating metrics. Negative behaviour may indicate an issue in the user interface or with the navigation. However, after you have identified the issue it is much easier to take steps to fix it.

5. USE EXISTING STANDARDS

While you can be creative in working out metrics that are specific to your company or product, the principle ideas of usability should still be taken into consideration. For instance, you could begin with the three ISO standards for usability—effectiveness, efficiency and satisfaction. Another good way to start is by using traditional 5Es as a basis. Obviously, you do not have to feel restricted by standard and existing metrics. However, you can use them as inspiration in creating a personal metrics system.

 
WHAT SHOULD BE MEASURED?

Having discussed the various criteria that should be taken into account upon creating metrics, let us now observe the metrics more closely. For each of the five E-categories, we have provided examples that can be used for inspiration. A short description of each metric has also been added.

EFFECTIVENESS METRICS

-  Number of completed tasks: Lets you know how many tasks or processes have been completed by the user.

-  Search form inefficiency: Identifies search forms that have been used to find something with no results.

-  Wrong default value of form: Identifies forms with constantly changing default values.

-  Number of navigation errors: Lets you know when the user has possibly got lost within the system, indicated by the user fruitlessly moving back-and-forth within the system.

-  Number of unnecessary interactions: Identifies the places where users have to make more clicks than necessary to get the desired result.

EFFICIENCY METRICS

-  Task completion speed: Indicates how long it takes for the user to complete a task or process. The less effort is required from the user, the better.

-  Task completion speed in comparison to expert users: Indicates how long it takes for ordinary or new users to complete a task in comparison to a user with expert knowledge of the system.

 
ENGAGEMENT METRICS

-  Visit frequency: Indicates how many users are on the web page and also helps to identify how many of them are new or returning.

-  Most recent visit: Indicates the last time a user visited the page and helps to determine when the page will be visited again (i.e., due to an advertisement).

-  Satisfaction metrics: If users are more satisfied, it is more likely that they will use your system or visit your web page again. Surveys are often conducted to measure user satisfaction. We recommend using the System Usability Scale or SUS, for instance.

ERROR TOLERANCE METRICS

-  Number of times users have sought IT support: Lets you know how often users seek IT support. The metric indicates whether or not the error was serious enough that the user was not able to proceed without IT support.

-  Number of error messages (including validation errors): Reveals how often users are confronted with errors and which elements are related to these errors.

-  Nonworking buttons and links: Helps to identify the elements that are not working the way users expect them to. Lets you know if a particular element is not explanatory enough (i.e., the button does not appear to be clickable and the user takes no notice of it).

EASE OF LEARNING METRICS

-  How quickly users reach the optimal speed: Lets you know if users are able to learn how to use your system quickly or if it takes them more time to practise.

-  Remembering previous times of use: Helps to determine if the user’s speed of completing the process increases or remains the same after repeated use of the system. The system has better learnability if users are able to familiarise themselves with the system more quickly.

-  Number of times help was used: Lets you know if users need help in order to move forward in the process. Helps to determine if a user interface needs more explanations because users have frequently used help feature.