COVID-19 Update: We are currently shipping orders daily. However, due to transit disruptions in some geographies, deliveries may be delayed. To provide all customers with timely access to content, we are offering 50% off Science and Technology Print & eBook bundle options. Terms & conditions.
Agile Data Warehousing for the Enterprise - 1st Edition - ISBN: 9780123964649, 9780123965189

Agile Data Warehousing for the Enterprise

1st Edition

A Guide for Solution Architects and Project Leaders

Author: Ralph Hughes
Paperback ISBN: 9780123964649
eBook ISBN: 9780123965189
Imprint: Morgan Kaufmann
Published Date: 8th October 2015
Page Count: 562
Sales tax will be calculated at check-out Price includes VAT/GST
Price includes VAT/GST

Institutional Subscription

Secure Checkout

Personal information is secured with SSL technology.

Free Shipping

Free global shipping
No minimum order.


Building upon his earlier book that detailed agile data warehousing programming techniques for the Scrum master, Ralph's latest work illustrates the agile interpretations of the remaining software engineering disciplines:

  • Requirements management benefits from streamlined templates that not only define projects quickly, but ensure nothing essential is overlooked.
  • Data engineering receives two new "hyper modeling" techniques, yielding data warehouses that can be easily adapted when requirements change without having to invest in ruinously expensive data-conversion programs. 
  • Quality assurance advances with not only a stereoscopic top-down and bottom-up planning method, but also the incorporation of the latest in automated test engines. 

Use this step-by-step guide to deepen your own application development skills through self-study, show your teammates the world's fastest and most reliable techniques for creating business intelligence systems, or ensure that the IT department working for you is building your next decision support system the right way.

Key Features

  • Learn how to quickly define scope and architecture before programming starts
  • Includes techniques of process and data engineering that enable iterative and incremental delivery
  • Demonstrates how to plan and execute quality assurance plans and includes a guide to continuous integration and automated regression testing
  • Presents program management strategies for coordinating multiple agile data mart projects so that over time an enterprise data warehouse emerges
  • Use the provided 120-day road map to establish a robust, agile data warehousing program


data warehousing professionals including architects, designers, data modelers, testers, database administrators, and project managers as well as IT managers, directors, and VPs

Table of Contents

  • Chapter 1. Solving Enterprise Data Warehousing’s “Fundamental Problem”
    • The Agile Solution in a Nutshell
    • Five Legs to Stand Upon
    • The Agile EDW Alternative is Ready to Deploy
    • Defining a Baseline Method for Agile EDW
    • Plenty of Motivation to “Go Agile”
    • Structure of the Presentation Ahead
    • Summary
  • Part I: Summaries of Generic Agile Development Methods
    • Chapter 2. Primer on Agile Development Methods
      • Defining “Agile”
      • Agile Manifesto Values and Principles
      • Scrum in a Nutshell
      • Contributions from Extreme Programming
      • Summary
    • Chapter 3. Introduction to Alternative Iterative Methods
      • Lean Software Development
      • Kanban
      • The Hybrid “Scrumban” Approach
      • Rational Unified Process
      • Summary
    • Part I References
      • Chapter 2
  • Part II: Review of Fast EDW Coding and Risk Mitigation
    • Chapter 4. Essential DW/BI Background and Definitions
      • Primary Source for DW/BI Standards
      • Basic Business Terms
      • Data and Information Terms
      • Information Services Terms
      • Software Engineering Terms
      • Basic Architectural Concepts
      • Architectural Frameworks
      • Additional Data Warehousing Concepts
      • Traditional Project Management Terms
      • Summary
    • Chapter 5. Recap of Agile DW/BI Coding Practices
      • Iterative Coding Alone Significantly Improves BI Projects
      • New Roles for DW/BI Projects
      • 80/20 Specifications
      • Developer Stories
      • Current Estimates
      • Adding Techniques from Kanban
      • Evidence-Based Service Level Agreements
      • Proof that Agile DW/BI Works
      • Summary
    • Chapter 6. Eliminating Risk Through Nested Iterations
      • EDW Programs Slip into “231 Swamps”
      • Agile’s Fundamental Risk Mitigation Technique
      • Agile Edw’s Extended Risk Mitigation Techniques
      • Summary
    • Part II References
      • Chapter 4
  • Part III: Agile EDW Requirements Management
    • Chapter 7. Balancing between Two Extremes
      • Building the Case for Effective Requirements Management
      • Easy to Overinvest in Requirements Management
      • Reasons Not to Overinvest in Requirement Work
      • Agile’s Approach Centers on Balance
      • Two Intersecting Requirements Management Value Chains
      • Business Analysts Implicit in Two Project Lead Roles
      • Summary
    • Chapter 8. Redefining the Epic Stack to Enable Value Accounting
      • Toward a Robust Epic Decomposition Framework
      • Testing Whether Stories are Good Enough
      • Clarifying Everything with Value Accounting
      • Allocating Value Throughout an Epic Tree
      • Value Buildups by Environment Provide Motivation and Clarity
      • Summary
    • Chapter 9. Artifacts for the Generic Requirements Value Chain
      • Beware of Requirements Churn
      • User Modeling/Personas
      • End Users’ Hierarchy of Needs
      • Mind Maps and Fishbone Diagrams
      • Vision Boxes
      • Vision Statements
      • Product Roadmaps
      • Summary
    • Chapter 10. Artifacts for the Enterprise Requirements Value Chain
      • The Generic Value Chain Can Overlook Crucial Requirements
      • ERM as a Flexible RM Approach
      • Focusing on Enterprise Aspects of Project Requirements
      • Uncovering Project Goals with Sponsor’s Concept Briefing
      • Identifying Project Objectives with Stakeholder’s Requests
      • Sketching the Solution with a Vision Document
      • Segmenting the Project with Subrelease Overview
      • Providing Developer Guidance with Module Use Cases
      • Summary
    • Chapter 11. Intersecting Value Chains for a Stereoscopic Project Definition
      • Intersecting the Two Value Chains
      • Addressing Nonfunctional Requirements
      • Supporting the Organization’s Software Release Cycle
      • Techniques for the Elaboration Phase
      • Prioritizing Project Backlogs
      • Managing Incremental Precision
      • Effort Levels by Team Roles
      • Conquering Complex Business Rules with an Embedded Method
      • Interfacing with Project Governance
      • Not Returning to a Waterfall Approach
      • Summary
    • Part III References
      • Chapter 7
  • Part IV: Agile EDW Data Engineering
    • Chapter 12. Traditional Data Modeling Paradigms and Their Discontents
      • EDW at a Crossroads
      • Models, Architectures, and Paradigms
      • Normalization Basics
      • Generalization Basics
      • The Standard Approach and its Data Modeling Paradigms
      • The Traditional Integration Layer as a Challenged Concept
      • “Straight-To-Star” as a Controversial Alternative
      • Four Change Cases for Appraising a Data Modeling Paradigm
      • Summary
    • Chapter 13. Surface Solutions Using Data Virtualization and Big Data
      • Leveraging Shadow it
      • Faster Value Delivery with Data Virtualization
      • An Agile Role for Big Data
      • Summary
    • Chapter 14. Agile Integration Layers with Hyper Normalization
      • Hyper Normalization Hinges on “Ensemble Modeling”
      • Hyper Normalized Data Modeling Concepts
      • Reusable ETL Modules Accelerate New Development
      • Common Data Retrieval Challenges and Their Solutions
      • Re-Architecting the EDW for Hyper Normalization
      • Enabling Evolution of Existing EDW Components
      • HNF-Powered Agile Solutions
      • Evidence of Success
      • Summary
    • Chapter 15. Fully Agile EDW with Hyper Generalization
      • Hyper Generalization Involves a Mix of Modeling Strategies
      • HGF Enables Model-Driven Development and Fast Deliveries
      • Loading Data into the Hyper Generalized Integration Layer
      • Retrieving Information from a Hyper Generalized EDW
      • Model-Driven Evolution and Fast Adaptation
      • Supporting Derived Elements
      • Addressing Performance Concerns
      • Demonstrating Agility Through Four Change Cases
      • HGF-Powered Agile Solutions
      • Evidence of Success
      • Summary
    • Part IV References
      • Chapter 12
  • Part V: Agile EDW Quality Management Planning
    • Chapter 16. Why We Test and What Tests to Run
      • Why Test?
      • An Agile Approach to Quality Assurance
      • “What to Test?” Answered with Top-Down Planning
      • A 2×2 Planning Matrix for Top-Down Test Selection
      • “What to Test?” Answered Bottom-Up
      • Summary
    • Chapter 17. Designating Who, When, and Where
      • Who Shall Write the Tests?
      • When Should Teammates Perform Their QA Duties?
      • Where Should Teammates Perform Their QA Duties?
      • Key Quality Responsibilities by Team Role
      • The Overarching Duties of the System Tester
      • How Many Testers are Needed?
      • Summary
    • Chapter 18. Deciding How to Execute the Test Cases
      • Good Agile Quality Plans Involve Numerous Test Executions
      • Step 1: Update the Top-Down Plan
      • Step 2: Start Building the Parameter-Driven Widgets
      • Step 3: Plan Out the Test Data Sets
      • Step 4: Implement the Engine, Whether Manual or Automated
      • Step 5: Define the Project’s Set of Testing Aspects
      • Step 6: Build and Populate the Test Data Repository
      • Step 7: Quantify the Testing Objectives
      • Step 8: Begin Creating Test Cases
      • Step 9: Start Up the Engine
      • Step 10: Visualize Project Progress with Quality Assurance
      • Step 11: Document the Team’s Success
      • Summary
    • Part V References
      • Chapter 16
  • Part VI: Integrating the Pieces of the Agile EDW Method
    • Chapter 19. The Agile EDW Subrelease Cycle
      • Making the Release Cycle a Repeatable Process
      • Traditional Notions of Data Governance
      • The Agile EDW Subrelease Value Cycle
      • Centering the Value Cycle on Data Governance and Quality
      • Guiding the Agile EDW Transition
      • Summary
    • Part VI References
      • Chapter 19


No. of pages:
© Morgan Kaufmann 2016
8th October 2015
Morgan Kaufmann
Paperback ISBN:
eBook ISBN:

About the Author

Ralph Hughes

Ralph Hughes

Ralph Hughes, former DW/BI practice manager for a leading global systems integrator, has led numerous BI programs and projects for Fortune 500 companies in aerospace, government, telecom, and pharmaceuticals. A certified Scrum Master and a PMI Project Management Professional, he began developing an agile method for data warehouse 15 years ago, and was the first to publish books on the iterative solutions for business intelligence projects. He is a veteran trainer with the world's leading data warehouse institute and has instructed or coached over 1,000 BI professionals worldwide in the discipline of incremental delivery of large data management systems.

A frequent keynote speaker at business intelligence and data management events, he serves as a judge on emerging technologies award panels and program advisory committees of advanced technology conferences. He holds BA and MA degrees from Stanford University where he studied computer modeling and econometric forecasting. A co-inventor of Zuzena, the automated testing engine for data warehouses, he serves as Chief Systems Architect for Ceregenics and consults on agile projects internationally.

Affiliations and Expertise

former DW/BI practice manager for a leading global systems integrator, has led numerous BI programs and projects for Fortune 500 companies in aerospace, government, telecom, and pharmaceuticals


"Agile Data Warehousing, as outlined in Ralph’s multiple books, has transformed the way our BI team manages projects.  In just two years, we are delivering releases in iterations three times more frequently on five times the number of concurrent projects, while improving application quality and the enterprise nature of our solutions." --Mark Giesbrecht, Senior Manager, Business Intelligence, major Canadian railroad

"Agile Data Warehouse for the Enterprise is a must read for any data professional tasked with delivering enterprise reporting and analytics in the nimble, speed-to-value environment that we find ourselves today.  It marries agile methodology with data warehousing best practices to create a blueprint to deliver value fast." --Nik Green, Director of Business Intelligence for multi-billion dollar food retailer

"This comprehensive guide provides a solid and complete foundation for agile EDW development including revolutionary new paradigms.  I especially like the ‘out of the box’ thinking, practicality of the techniques, and the research and case studies backing up the validity of the proposed approaches.  Recommend reading for all DW/BI technical leaders." --Len Silverston, CEO of Universal Data Models and author of the Data Model Resource Book series

"If you’re still programming data warehouses by hand, you're wasting 90 percent of your time and money.  Ralph’s guide for project leaders not only outlines the automated, metadata-driven development we've been practicing for years in the Netherlands, but also links that practice to agile requirements, coding, and quality assurance.  It is a unique book, a must read." --Ronald Damhof, DW/BI Consultant to the Dutch Central Bank

"The industry has struggled to bring the mechanics and benefits of Agile to the data warehousing and business intelligence communities. Ralph’s works in this area is timely and important to assist in driving success within your enterprise and project teams. Following the guidance found in this book will help you deliver value at a rapid pace." --Tom Hammergren, President of Balanced Insight and author of Data Warehousing: Building the Corporate Knowledge Base

"Ralph has customized the agile development methodology for the unique needs of business intelligence.  My project teams were able to quickly understand and apply his approach without losing time to adapting generic Scrum techniques to the rigors of data integration and analytics." --Ron Lewis, program manager at a Fortune 1000 financial services company

"As a friend of Ralph and fellow Agile DW/BI author, I am very excited about this book. Ralph’s extensive research into what works; his hands-on application of those techniques; and his well-organized method of presenting information, are combined in this game changing book for data warehouse practitioners seeking greater agility." --Ken Collier PhD, author of Agile Analytics, Addison-Wesley, 2011

"This book presents a thorough approach to building quality into enterprise data warehouses, mitigating risk by applying proven models for testing, and detailing alternatives so that you can experiment to find the alternatives that work best for your team. These aren’t  mere off the cuff ideas, but proven techniques to overcoming the huge challenges of EDW by applying agile principles, illustrated with stories from the trenches." --Lisa Crispin, Co-Author of More Agile Testing: Learning Journeys for the Whole Team, and Agile Testing: A Practical Guide for Testers and Agile Teams

"“Ralph’s notion of ‘80/20 specifications’ for data warehousing projects really worked, saving our business partners the pain of doing an exhaustive requirements specification up-front.  This approach got the team developing the most important features first, letting the product owner fill in details on topics as each one came up during development." --Naveen Thalanki, Project Manager for a Fortune 500 company

"Agile Data Warehousing for the Enterprise is a “how to” book with innovative method and process components such as hyper data modeling and an iterative sub-release value cycle.  Ralph provides a clear outline of the concepts, methods, and frameworks you'll need to assemble a world-class BI/DW program of your own." --Hans Hultgren, author of Modeling the Agile Data Warehouse with Data Vault and BI/DW industry advisor

"One of TDWI’s most popular faculty members, Ralph packs this book with detailed direction and experience-driven insights about how to overcome the most challenging aspects of agile for business intelligence and data warehousing applications.  He has become an important thought leader for organizations seeking to increase the agility, quality, and ROI of their BI/DW programs." --David Stodder, Director of Research for Business Intelligence, The Data Warehousing Institute

"Ralph's book goes way beyond just agile programming—it illustrates an iterative approach to the full development life cycle and is particularly relevant to issues of data quality that we focus on at DAMA.  The hyper modeling techniques in particular will allow teams to avoid the death trap of producing big, risky application designs up-front before a project's requirements are fully known." -- Ken Dunn, President of DAMA’s Houston chapter

"Agile techniques and Ralph’s adaptations for enterprise data warehousing have dramatically improved our ability to understand business needs and to then plan, track, and deliver upon those needs for large and small projects. This book provides a thorough treatment of the vital leadership practices for agile projects including requirements, architecture, and quality assurance." -- Richard Tench, Manager, Information Delivery, major Canadian insurance company

Ratings and Reviews