Measuring the User Experience

Collecting, Analyzing, and Presenting Usability Metrics


  • Thomas Tullis, Senior Vice President of User Experience, Fidelity Investments, USA
  • William Albert, Director, Design and Usability Center, Bentley University, USA

Effectively measuring the usability of any product requires choosing the right metric, applying it, and effectively using the information it reveals. Measuring the User Experience provides the first single source of practical information to enable usability professionals and product developers to do just that. Authors Tullis and Albert organize dozens of metrics into six categories: performance, issues-based, self-reported, web navigation, derived, and behavioral/physiological. They explore each metric, considering best methods for collecting, analyzing, and presenting the data. They provide step-by-step guidance for measuring the usability of any type of product using any type of technology.
View full description


Usability professionals, developers, programmers, information architects, interaction designers, market researchers, and students in an HCI or HFE program.


Book information

  • Published: March 2008
  • ISBN: 978-0-12-373558-4


"If Tom and Bill could convince me, perhaps the world’s biggest fan of qualitative testing, that usability metrics are really valuable-which they have, in this wonderful book-then there’s no doubt they’ll convince you. I loved reading this book, because it was exactly like having a fascinating conversation with a very smart, very seasoned, and very articulate practitioner. They tell you everything you need to know (and no more) about all the most useful usability metrics, explain the pros and cons of each one (with remarkable clarity and economy), and then reveal exactly how they actually use them after years and years of real world experience. Invaluable!" Steve Krug, author of Don’t Make Me Think: A Common Sense Approach to Web Usability "This book is a great resource about the many ways you can gather usability metrics without busting your budget. If you’re ready to take your user experience career to the next level of professionalism, Tullis and Albert are here for you and share generously of their vast experience. Highly recommended." Jakob Nielsen, Principal, Nielsen Norman Group, author of Usability Engineering and Eyetracking Web Usability "If you do any type of usability testing, you need this book. Tullis and Albert have written a clear and comprehensive guide with a common-sense approach to usability metrics." Ginny Redish, President of Redish and Associates, Inc., author of Letting Go of the Words

Table of Contents

CHAPTER 1 Introduction1.1 Organization of This Book1.2 What Is Usability? 1.3 Why Does Usability Matter? 1.4 What Are Usability Metrics? 1.5 The Value of Usability Metrics 1.6 Ten Common Myths about Usability MetricsCHAPTER 2 Background2.1 Designing a Usability Study2.2 Types of Data 2.3 Metrics and Data2.4 Descriptive Statistics 2.5 Comparing Means 2.6 Relationships between Variables2.7 Nonparametric Tests 2.8 Presenting Your Data Graphically2.9 SummaryCHAPTER 3 Planning a Usability Study3.1 Study Goals3.2 User Goals3.3 Choosing the Right Metrics: Ten Types of Usability Studies3.4 Other Study Details 3.5 SummaryCHAPTER 4 Performance Metrics4.1 Task Success4.2 Time-on-Task4.3 Errors 4.4 Efficiency 4.5 Learnability 4.6 SummaryCHAPTER 5 Issues-Based Metrics5.1 Identifying Usability Issues5.2 What Is a Usability Issue?5.3 How to Identify an Issue5.3.1 In-Person Studies5.4 Severity Ratings 5.5 Analyzing and Reporting Metrics for Usability Issues5.6 Consistency in Identifying Usability Issues5.7 Bias in Identifying Usability Issues5.8 Number of Participants5.9 SummaryCHAPTER 6 Self-Reported Metrics6.1 Importance of Self-Reported Data6.2 Collecting Self-Reported Data6.3 Post-Task Ratings6.4 Post-Session Ratings6.5 Using SUS to Compare Designs6.6 Online Services6.7 Other Types of Self-Reported Metrics 6.8 SummaryCHAPTER 7 Behavioral and Physiological Metrics 7.1 Observing and Coding Overt Behaviors7.2 Behaviors Requiring Equipment to Capture 7.3 SummaryCHAPTER 8 Combined and Comparative Metrics8.1 Single Usability Scores8.2 Usability Scorecards 8.3 Comparison to Goals and Expert Performance8.4 SummaryCHAPTER 9 Special Topics9.1 Live Website Data9.2 Card-Sorting Data 9.3 Accessibility Data 9.4 Return-on-Investment Data9.5 Six Sigma9.6 SummaryCHAPTER 10 Case Studies10.1 Redesigning a Website Cheaply and Quickly, Hoa Loranger10.2 Usability Evaluation of a Speech Recognition IVR, James R. Lewis10.3 Redesign of the Website Robert Bailey, Cari Wolfson, and Janice Nall 10.4 Usability Benchmarking: Mobile Music and Video, Scott Weiss and Chris Whitby10.5 Measuring the Effects of Drug Label Design and Similarity on Pharmacists’ Performance, Agnieszka Bojko10.6 Making Metrics Matter, Todd Zazelenchuk 10.6.7 Acknowledgment10.6.8 Biography 10.6.9 ReferencesCHAPTER 11 Moving Forward11.1 Sell Usability and the Power of Metrics 11.2 Start Small and Work Your Way Up11.3 Make Sure You Have the Time and Money 11.4 Plan Early and Often .11.5 Benchmark Your Products .11.6 Explore Your Data 11.7 Speak the Language of Business11.8 Show Your Confidence11.9 Don’t Misuse Metrics 11.10 Simplify Your Presentation