CUDA Application Design and Development

1st Edition

Authors: Rob Farber
Paperback ISBN: 9780123884268
eBook ISBN: 9780123884329
Imprint: Morgan Kaufmann
Published Date: 31st October 2011
Page Count: 336
Tax/VAT will be calculated at check-out
Compatible Not compatible
VitalSource PC, Mac, iPhone & iPad Amazon Kindle eReader
ePub & PDF Apple & PC desktop. Mobile devices (Apple & Android) Amazon Kindle eReader
Mobi Amazon Kindle eReader Anything else

Institutional Access


As the computer industry retools to leverage massively parallel graphics processing units (GPUs), this book is designed to meet the needs of working software developers who need to understand GPU programming with CUDA and increase efficiency in their projects. CUDA Application Design and Development starts with an introduction to parallel computing concepts for readers with no previous parallel experience, and focuses on issues of immediate importance to working software developers: achieving high performance, maintaining competitiveness, analyzing CUDA benefits versus costs, and determining application lifespan.

The book then details the thought behind CUDA and teaches how to create, analyze, and debug CUDA applications. Throughout, the focus is on software engineering issues: how to use CUDA in the context of existing application code, with existing compilers, languages, software tools, and industry-standard API libraries.

Using an approach refined in a series of well-received articles at Dr Dobb's Journal, author Rob Farber takes the reader step-by-step from fundamentals to implementation, moving from language theory to practical coding.

Key Features

  • Includes multiple examples building from simple to more complex applications in four key areas: machine learning, visualization, vision recognition, and mobile computing
  • Addresses the foundational issues for CUDA development: multi-threaded programming and the different memory hierarchy
  • Includes teaching chapters designed to give a full understanding of CUDA tools, techniques and structure.
  • Presents CUDA techniques in the context of the hardware they are implemented on as well as other styles of programming that will help readers bridge into the new material


Software engineers, programmers, hardware engineers, advanced students

Table of Contents

CHAPTER 1 First Programs and How to Think in CUDA

Source Code and Wiki

Distinguishing CUDA from Conventional Programming with a Simple Example

Choosing a CUDA API

Some Basic CUDA Concepts

Understanding Our First Runtime Kernel

Three Rules of GPGPU Programming

Big-O Considerations and Data Transfers

CUDA and Amdahl’s Law

Data and Task Parallelism

Hybrid Execution: Using Both CPU and GPU Resources

Regression Testing and Accuracy

Silent Errors

Introduction to Debugging

UNIX Debugging

Windows Debugging with Parallel Nsight


CHAPTER 2 CUDA for Machine Learning and Optimization

Modeling and Simulation

Machine Learning and Neural Networks

XOR: An Important Nonlinear Machine-Learning Problem

Performance Results on XOR

Performance Discussion


The C++ Nelder-Mead Template

CHAPTER 3 The CUDA Tool Suite: Profiling a PCA/NLPCA



Obtaining Basic Profile Information

Gprof: A Common UNIX Profiler

The NVIDIA Visual Profiler: Computeprof

Parallel Nsight for Microsoft Visual Studio

Tuning and Analysis Utilities (TAU)


CHAPTER 4 The CUDA Execution Model

GPU Architecture Overview

Warp Scheduling and TLP

ILP: Higher Performance at Lower Occupancy

Little’s Law

CUDA Tools to Identify Limiting Factors



The CUDA Memory Hierarchy

GPU Memory

L2 Cache

L1 Cache

CUDA Memory Types

Global Memory


CHAPTER 6 Efficiently Using GPU Memory


Utilizing Irregular Data Structures

Sparse Matrices and the CUSP Library

Graph Algorithms

SoA, AoS, and Other Stru


No. of pages:
© Morgan Kaufmann 2012
Morgan Kaufmann
eBook ISBN:
Paperback ISBN:

About the Author

Rob Farber

Rob Farber has served as a scientist in Europe at the Irish Center for High-End Computing as well as U.S. national labs in Los Alamos, Berkeley, and the Pacific Northwest. He has also been on the external faculty at the Santa Fe Institute, consultant to fortune 100 companies, and co-founder of two computational startups that achieved liquidity events. He is the author of “CUDA Application Design and Development” as well as numerous articles and tutorials that have appeared in Dr. Dobb's Journal and Scientific Computing, The Code Project and others.

Affiliations and Expertise

CEO/Publisher of, Wall Street Analyst, and consultant to scientific and commercial technology companies around the world.


The book by Rob Faber on CUDA Application Design and Development is required reading for anyone who wants to understand and efficiently program CUDA for scientific and visual programming. It provides a hands-on exposure to the details in a readable and easy to understand form. Jack Dongarra, Innovative Computing Laboratory, EECS Department, University of Tennessee

GPUs have the potential to take computational simulations to new levels of scale and detail. Many scientists are already realising these benefits, tackling larger and more complex problems that are not feasible on conventional CPU-based systems. This book provides the tools and techniques for anyone wishing to join these pioneers, in an accessible though thorough text that a budding CUDA programmer would do well to keep close to hand. Dr. George Beckett, EPCC, University of Edinburgh

With his book, Farber takes us on a journey to the exciting world of programming multi-core processor machines with CUDA. Farber's pragmatic approach is effective in guiding the reader across challenges and their solutions.   Farber's broader presentation of parallel programming with CUDA ranging from CUDA in Cloud and Cluster environments to CUDA for real problems and applications helps the reader learning about the unique opportunities this parallel programming language can offer to the scientific community. This book is definitely a must for students, teachers, and developers! Michela Taufer, Assistant Professor, Department of Computer and Information Sciences, University of Delaware

Rob Farber has written an enlightening and accessible book on the application to CUDA for real research tasks, with an eye to developing scalable and distributed GPU applications.  He supplies clear and usable code examples combined with insight about _why_ one should use a particular approach.  This is an excellent book filled with practical advice for experienc