Objectives

• The main goals of the chapter are to accomplish the following:

• Explain how to do usability testing.

• Describe how different types of studies are being done remotely.

• Outline the basics of experimental design.

• Describe how to do in the wild studies.

Introduction

Imagine that you have designed a new app to allow school children ages 9 or 10 years old and their parents to share caring for the class hamster over the school holidays. The app will schedule which children are responsible for the hamster and when, and it will record when it is fed. The app will also provide detailed instructions about when the hamster is scheduled to go to another family and the arrangements about when and where it will be handed over. In addition, both teachers and parents will be able to access the schedule and send and leave messages for each other. How would you find out whether the children, their teacher, and their parents can use the app effectively and whether it is satisfying to use? What evaluation methods would you employ?

In this chapter, we describe evaluation studies that take place in a spectrum of settings, from controlled laboratories to in-the- wild studies (sometimes referred to as natural settings or field studies), and from face-to-face situations to remote settings using digital technologies.

Increasingly, elements of usability testing, experiments, and in-the-wild studies are conducted remotely using tracking software to record participants’ interactions and using social media for communication with participants. As mentioned in Chapter 14, remote usability started in the 1990s (e.g., Hartson et al., 1998) and became a key alternative during the COVID-19 pandemic when social distancing was necessary to protect people from catching the virus (Gov.UK, 2021; Center for Digital Public Services, 2021). In addition to keeping both participants and evaluators safe, remote testing is attractive because it saves time traveling to distant locations and the need to schedule and pay for lab time. Therefore, remote testing tends to be less expensive than in-person testing. But there are also some drawbacks to be aware of (Moran, 2021). For example, remote testing may require more careful advance planning as it will be more difficult for the evaluator to address problems. Developing a relationship with remote participants may also be more difficult and take longer.

Within this range of user testing approaches we focus on the following:

• Usability testing, which takes place in usability labs and other controlled lab-like settings

• Remote studies, in which evaluators may interact with participants synchronously, asynchronously, or not at all

• Experiments, which usually take place in research labs but can sometimes be done remotely

• In-the-wild studies, which take place in natural settings, such as people’s homes, schools, work, and leisure environments.