Chapter 13: An Evaluation Framework
| Web Resources
| Assignment Comments
| Teaching Materials
The main aims of this chapter are to:
- Introduce and explain the DECIDE framework.
- Discuss the conceptual, practical, and ethical issues involved in evaluation.
As products evolve from initial ideas through conceptual design and prototypes, iterative
cycles of design and evaluation help to ensure that they meet users' needs. However, deciding
when and how to evaluate a product requires careful consideration and may be different for
different kinds of products. This can be challenging given the rapid development of a wide
variety of mobile and distributed systems.
The design process starts with the designers working to develop a product that meets
users' requirements, but understanding requirements tends to happen by a process of negotiation
between designers and users over a period of time. As designers understand users'
needs better, their designs reflect this understanding. Similarly, as users see and experience
design ideas, they are able to give better feedback that enables the designers to improve their
designs. The process is cyclical, with evaluation facilitating understanding between designers
Evaluation is driven by questions about how well the design or particular aspects of it
satisfy requirements and offer appropriate user experiences. Some of these questions provide
high-level goals to guide the evaluation. For example, does this product excite users so that
they will buy and use it? Others are much more specific. Can users find a particular menu
item? Do they interpret a particular graphic as the designers intended and do they fi nd it
attractive? Practical constraints play a big role in shaping how evaluation is done: tight
schedules, low budgets, or limited access to users constrain what evaluators can do and may
prompt them to seek new ways of doing evaluations, e.g. using crowdsourcing as discussed
in Chapter 12. There are ethical considerations too: medical records are confi dential, certain areas of people's homes are private, and so is some of the information that users put on their
Facebook and other social networking sites.
There is a broad repertoire of evaluation methods that can be tailored for specifi c circumstances,
and experienced designers get to know what works and what doesn't. When
planning evaluations, evaluators must consider the nature of each product, the target user
population, where the product will be used and the contexts of use, as well as logistical
issues. Planning evaluation studies involves asking questions about the process and anticipating
potential problems. Within interaction design there are many books and websites that
list different techniques and guidelines for conducting an evaluation, but there is very little
overarching guidance for how to plan an evaluation. To help you, we propose the DECIDE
framework, which provides a structure for planning evaluation studies.