Evaluation of diagnostic systems

Evaluation of diagnostic systems

1st Edition - January 28, 1982

Write a review

  • Author: John Swets
  • eBook ISBN: 9780323141642

Purchase options

Purchase options
DRM-free (PDF)
Sales tax will be calculated at check-out

Institutional Subscription

Free Global Shipping
No minimum order


Evaluation of Diagnostic Systems: Methods from Signal Detection Theory addresses the many issues that arise in evaluating the performance of a diagnostic system, across the wide range of settings in which such systems are used. These settings include clinical medicine, industrial quality control, environmental monitoring and investigation, machine and metals inspection, military monitoring, information retrieval, and crime investigation. The book is divided into three parts encompassing 11 chapters that emphasize the interpretation of diagnostic visual images by human observers. The first part of the book describes quantitative methods for measuring the accuracy of a system and the statistical techniques for drawing inferences from performance tests. The subsequent part covers study design and includes a detailed description of the form and conduct of an image-interpretation test. The concluding part examines the case study of a medical imaging system that serves as an example of both simple and complex applications. In this part, three mammographic modalities are used: industrial film radiography, low-dose film radiography, and xeroradiography. The case study focuses on the overall reliability of accuracy indices made by its main components, that is, the variabilities across cases, across readers, and within individual readers. The supplementary texts provide study protocols, a computer program for processing test results, and an extensive list of references that will assist the reader in applying those evaluative methods to diagnostic systems in any setting. This book is of value to scientists and engineers, as well as to applied, quantitative, or experimental psychologists who are engaged in the study of the human processes of discrimination and decision making in either perceptual or cognitive tasks.

Table of Contents

  • Preface



    The General Setting

    The General Problem

    The Proposed Solution

    Four Types of Evaluation

    General Issues in Test Design

    Some Application Vignettes

    Organization of This Book

    Part I Quantitative Methods

    Chapter 1 Fundamentals of Accuracy Analysis

    1.1 Two Aspects of Performance

    1.2 Analysis of Performance via the ROC Curve

    1.3 Often Used, but Inadequate, Indices of Accuracy

    1.4 Recommended Indices of Accuracy

    1.5 A Computational Example

    1.6 A Suitable Accuracy Index Based on One ROC Point

    1.7 Indices of the Decision Criterion

    1.8 Summary

    Chapter 2 Extensions of Accuracy Analysis

    2.1 Multiple Alternatives

    2.2 Other Interpretations of ROC Data

    2.3 Combining Individual Results into a Group or Average ROC

    2.4 Summary

    Chapter 3 Statistical Design of a Performance Test

    3.1 Questions to Be Answered by the Reading Test

    3.2 Logistic and Cost Constraints

    3.3 General Approach to Statistical Analysis of Accuracy Data

    3.4 The Degree of Statistical Power Required

    3.5 Some Practical Rules for Study Design

    3.6 Summary

    Chapter 4 Statistical Treatment of Accuracy Results

    4.1 Standard Error of the Accuracy Index

    4.2 Confidence Interval of the Accuracy Index

    4.3 Statistical Significance of a Difference between Modalities

    4.4 Statistical Analysis of True-Positive Probability at a Fixed False-Positive Probability

    4.5 Assumptions Underlying the Recommended Statistical Procedure

    4.6 Tests Comparing More Than Two Modalities

    4.7 Summary

    Chapter 5 Forms of Efficacy Analysis

    5.1 Cost-Benefit Analysis: A Conceptual Starting Point

    5.2 Impediments to Cost-Benefit Analysis

    5.3 Tailoring the Analysis to Suit the Situation

    5.4 Setting the Efficacy Analysis at an Appropriate Level

    5.5 Summary

    Part II Experimental Methods

    Chapter 6 Elements of Study Design

    6.1 Developing Study Goals

    6.2 Outlining the End Product

    6.3 Planning the Collection of Data

    6.4 Summary

    Chapter 7 Procedure of a Performance Test

    7.1 Case and Image Presentation

    7.2 Response Format

    7.3 Background Information Given Readers on Each Case

    7.4 Timing of Trials and Sessions

    7.5 Viewing Environment

    7.6 Role of Test Administrator

    7.7 Reader Orientation

    7.8 Feedback to and from Readers

    7.9 Summary

    Chapter 8 Drawing the Original Case Sample

    8.1 Functions of the Original Case Sample

    8.2 Types of Data Collected on Each Case

    8.3 Selecting a Representative Case Sample

    8.4 Bias in Case Data

    8.5 Required Size of the Original Case Sample

    8.6 Summary

    Chapter 9 Selection of Test Cases

    9.1 Number of Cases for Adequate Representation

    9.2 Proportional Representation of Case Types

    9.3 Screening Out Inadequate Cases

    9.4 Checking Representativeness of the Reading-Test Sample

    9.5 Summary

    Chapter 10 Selection of Test Readers

    10.1 Number of Readers for Adequate Representation

    10.2 Type of Reader

    10.3 Summary

    Part III A Methodological Case Study

    Chapter 11 Illustrative Evaluation of Imaging Systems

    11.1 Case Selection

    11.2 Response Format

    11.3 Experimental Design

    11.4 Test Procedures

    11.5 Reader Ratings of Image Quality

    11.6 Accuracy Results

    11.7 Correlational Analyses

    11.8 Obtaining Estimates of Pure Variance and Correlation Terms

    11.9 Discussion: Sizing a Final Study from a Pilot Study


    Appendix A Sequence of Steps in Evaluation

    Appendix B Representativeness of a Performance-Test Sample

    Appendix C Illustrative Posttest Questionnaire

    Appendix D Computer Program for Accuracy Analysis

    Appendix E References to ROC Applications



Product details

  • No. of pages: 270
  • Language: English
  • Copyright: © Academic Press 1982
  • Published: January 28, 1982
  • Imprint: Academic Press
  • eBook ISBN: 9780323141642

About the Author

John Swets

Ratings and Reviews

Write a review

There are currently no reviews for "Evaluation of diagnostic systems"