Usability Testing – Ananya Pandya

User Testing

Learn how to set up user study and analyze user preferences, and receive feedback to do the next iteration of prototyping.

Explore Now

Overview

A virtual testing ground for your Autonomous vehicle. Realsynth demo contains different real-world scenarios. A tutorial for the demo is included in the user interface guiding you in with the help of the quest. This comprehensive redesign offers Gamified tutorials, an intuitive user interface, a high degree of customizability, and effective sales value.

Goals

To guide the development of the user study we decided to look to the list of requirements that we initially defined in the need-finding phase. This, in conjunction with a list of features prioritized by the stakeholder, formed the basis for the research questions we sought to answer in the user study.

Role

UX Designer -
ETH Zürich

Area

Prototype Testing,
Usability Testing,

Team

2 students

Duration

Nov - Dec 2018

Client

Designing User Study

We followed a standard procedure for the study. At first, we defined basic key parameters to separate potential novice and more expert users. Then we introduced them to the company and showed them a video that was intended to demonstrate the key value of the full product (although not the features). In a subsequent step, the participants performed different tasks or completed questionnaires, that covered the research questions that we initially defined.

Each task is connected to one or more research question. We set clear metrics to be able to properly evaluate the research questions. We will explain the tasks in detail later in the report and directly introduce the results as to maintain the connection between the research questions and the corresponding results.

Setup

We guided the participant through the different tasks using a Google Form questionnaire.To perform the various tasks the user was introduced to a Med-Fi Prototype:

Task 0 - Demographic Questionnaire

This task collects basic demographic data, as well as the subject’s level of familiarity with the computers, autonomous vehicles, and the company in order to evaluate the characteristics of the population sample and investigate the presence of possible bias.

Goal: Find out the level of experience with Autonomous Driving/Computers

Metric: Self-assessment using a Likert Scale.

Task 1 - RQ - How popular is the tutorial?
How do constraints influence popularity?

In the first task, we wanted to assess the users' need for a tutorial (i.e. if we should offer one) and we wanted to find out how detailed the tutorial should be (i.e. with what granularity the demo should be explored).

Goal: Investigate how popular the use of a tutorial is among users and the characteristics that it should have

Hypothesis: Users are more likely to complete a tutorial only as long as the number of quests and time to complete remain limited.

Metric: Numerical Question

Task 2 - RQ - How effective is the quest-based tutorial?

We designed this task to find out whether a user would be more efficient in completing a certain action in the demo if they had completed the tutorial containing similar tasks beforehand compared to if they had to explore the demo on their own.

The experimenter would instruct the participant on the task and then measure and note the time as well as the number of clicks it took for the participant to complete an action. It was randomly decided whether a user performed this with or without a tutorial.

Goal: We want to measure how effective user using the tutorial are comparing to the user who freely explores

Metrics:

Number of Clicks
Time to Completion
Subjective Questionnaire
- On effectiveness
- On interest in the tutorial

Task 3 - RQ - How fast/easy to navigate through?

This task was designed to find out whether the clustering/menu icons of the different items would enable the user to quickly locate the action/item they were looking for.
The participants were asked to complete two actions, which were not part of the demo or other tasks, and the experimenter recorded the number of clicks as well as the time it took the participant to complete this action. This task was randomly performed before or after Task 2 to counterbalance a potential learning effect.

Goal: Find out if the clustering of features is intuitive for the user.

Metrics:

Time to completion
Number of Clicks
Subjective Questionnaire
- on the feeling of intuitiveness
- on communication clarity
- on visual clarity

Task 4 - RQ - Is the demo better than the current version?
In What way?

After having shown the current demo and the Hi-Fi MockUp of our prototype to the participants, they were asked to fill a questionnaire comparing the two demos, in order to evaluate if an improvement was made.

Goal: Comparison with original demo to see if an improvement was made.

Metrics:

Questionnaire on subjective comparison of
- Speed
- Intuitiveness/clarity
- Sales value

Results

In total, a sum of 6 tasks was designed for usability testing. In this phase, we tried to include all other associations that participants had with our prototype. The new design concept is regarded as more clean and also faster to operate. People also tended to prefer to present the new version.

Comparison of task completion time between the participants that performed the tutorial and the ones who did not. The two investigated tasks were: “change the time of day” and “Rotate the rear sensor”.

Most of the participants said that it should look more realistic, which was already covered in earlier tasks. The other comments such as: "Small Menu/Large screen", "too many clicks if slide-over menu" will be addressed in the iteration process of prototyping and shall be fulfilled in high fidelity prototype.

Key Takeaways

Due to the time limitations introduced by the end of the semester, we will conduct one more iteration of our prototype. We will most likely gather subjective feedback on the changed features. Due to several limitations of the quantitative results, we rely more on qualitative feedback to define the next steps.

We should introduce 3-5 quests based on the following items (deduced from complexity and prioritization):

1. Changing Object Properties of an existing object
2. Changing the segmentation view
3. Adapting a sensor property
4. Providing steering input via the command panel

The tutorial should also not provide too much guidance to the user.

Furthermore, we need to switch to icons that are more intuitive. This is something that we will re-evaluate with the participants once completed.
We will also add labels (on mouse-over) to the menu in order to achieve the ideal balance between visually unpleasing text labels and the clearest communication method.
The next prototype will be colorized to convey a more final version, addressing the feedback that it looked like a sketch or that it was visually raw or abstract.