How to measure usability (Part III: building a metric system)

19 Feb 2019

Liisa Prits

In the second part of this series, we gave advice on what to keep in mind upon creating metrics and how to start creating a system. In the final part we discuss metric formulas and calculations, explain how to interpret the results and what to do with the information.

HOW TO CALCULATE USABILITY SCORE?

In order to calculate usability score, you must first have the results for each metric. Although the formula may differ depending on the specific metric, the traditional approach to compiling and calculating metrics is that “something happened X number times”, which is then divided with the total number of all occurrences. Therefore, if we take the metric Number of times help was used, the formula would be as follows: (The number of times help was used / the total number of all uses) * 100.

Additional methods may be applied with more specific metrics, such as when you first need to define the normal level of something. The metric Number of unnecessary interactions is one of such occasions when you should first determine the optimal number of interactions in order to compare the results.

Next you should calculate the arithmetic average of all the results of the metrics based on category. This means that the average learnability rate is calculated based on the metrics contained within that category.

Lastly, you need to calculate the final usability score by adding the average rates of all the categories and dividing the result with the total number of categories. If you have used the 5E categories, every category makes up 20% of the end result.

INTERPRETING THE RESULTS

The results from the metrics are in percentages. The usability score is 100% when no negative situations have occurred. However, usually some negative situations do occur and in that case, the usability score may be interpreted as follows:

0–29% - information system usability is poor

30–69% - information system usability is average

70–100% - information system usability is good

All metrics are equal within their category and all categories influence the end result equally. If the usability score is low, you can figure out the issue by looking at the metrics of the specific category. For instance, let us assume your system’s efficiency metric gives very poor results. You can then figure out if the problem is related to a specific element of the user interface, whether the process is frequently discontinued or whether people have difficulties finding information.

Having found the source of the problem, you can start thinking about solutions. While some situations may be easier to solve and others may end up costlier (e.g., navigation must be redesigned), ultimately it will help you improve your system and offer a better experience for the users. Remember that a system with good usability will encourage users to return and spend their time on your web page.

WHAT ELSE SHOULD BE TAKEN INTO ACCOUNT?

When creating metrics and calculating the results, it is important to remember what each of the metrics represents and how they fit within the system. Since the entire system has been built upon percentages, the value of each measured element has to be converted into percentages as well.

The metrics are divided into positive and negative ones. Higher results for negative metrics indicate a bad result, while higher results for positive metrics indicate a good result.

For instance, getting 100% for the metric Wrong default value of form would be the worst possible result as it would indicate that the default value of the form has been changed on absolutely every instance.

However, getting 100% for the metric Number of completed tasks would be the best possible result and it would indicate that the user has completed all their tasks. Since there are both types of metrics in the system, the negative metrics must be converted using a formula (100 – the result of a negative metric in percentages).

Another important factor is the period of time during which these metrics have been applied. We recommend using all metrics as a single system during the same time period. The recommended period of time is one month, which is enough to collect the necessary data. Different time periods can be then compared to one another. It is especially useful to identify the effect of different versions, updates and ad campaigns on the final usability score.