Markov Processes for Stochastic Modeling

Markov Processes for Stochastic Modeling

2nd Edition - May 22, 2013

Write a review

  • Author: Oliver Ibe
  • Paperback ISBN: 9780323282956
  • eBook ISBN: 9780124078390

Purchase options

Purchase options
DRM-free (EPub, PDF, Mobi)
Sales tax will be calculated at check-out

Institutional Subscription

Free Global Shipping
No minimum order


Markov processes are processes that have limited memory. In particular, their dependence on the past is only through the previous state. They are used to model the behavior of many systems including communications systems, transportation networks, image segmentation and analysis, biological systems and DNA sequence analysis, random atomic motion and diffusion in physics, social mobility, population studies, epidemiology, animal and insect migration, queueing systems, resource management, dams, financial engineering, actuarial science, and decision systems. Covering a wide range of  areas of application of Markov processes, this second edition is revised to highlight the most important aspects as well as the most recent trends and applications of Markov processes. The author spent over 16 years in the industry before returning to academia, and he has applied many of the principles covered in this book in multiple research projects. Therefore, this is an applications-oriented book that also includes enough theory to provide a solid ground in the subject for the reader.

Key Features

  • Presents both the theory and applications of the different aspects of Markov processes
  • Includes numerous solved examples as well as detailed diagrams that make it easier to understand the principle being presented
  • Discusses different applications of hidden Markov models, such as DNA sequence analysis and speech analysis.


Graduate and upper-level undergraduate students, researchers and practitioners working in Markov Processes.

Table of Contents

  • Chapter 1: Basic Concepts

    Review of Probability

    Random Variables

    Transform Methods

    Bivariate Random Variables

    Many Random Variables

    Fubini’s Theorem

    Sums of Independent Random Variables

    Some Probability Distributions

    Introduction to Stochastic Processes

    Classification of Stochastic Processes

    Characterizing a Stochastic Process

    Stationary Stochastic Processes

    Ergodic Stochastic Processes

    Some Models of Stochastic Processes

    Chapter 2: Introduction to Markov Processes


    Structure of Markov Processes

    Strong Markov Property

    Applications of Discrete-time Markov Processes

    Applications of Continuous-time Markov Processes

    Applications of Continuous-state Markov Processes

    Chapter 3: Discrete-Time Markov Chains


    State Transition Probability Matrix

    State Transition Diagrams

    Classification of States

    Limiting-State Probabilities

    Sojourn Time

    Transient Analysis of Discrete-Time Markov Chains

    First Passage and Recurrence Times

    Occupancy Times

    Absorbing Markov Chains and the Fundamental Matrix

    Reversible Markov Chains

    Chapter 4: Continuous-Time Markov Chains


    Transient Analysis

    Birth and Death Processes

    First Passage Time

    The Uniformization Method

    Reversible Continuous-Time Markov Chains


    Chapter 5: Markovian Queueing Systems


    Description of a Queueing System

    The Kendall Notation

    The Little’s Formula

    The PASTA Property

    The M/M/1 Queueing System

    Examples of Other M/M Queueing Systems

    M/G/1 Queue

    G/M/1 Queue

    Chapter 6: Markov Renewal Processes

    Renewal Processes

    The Renewal Equation

    The Elementary Renewal Theorem

    Random Incidence and Residual Time

    Markov Renewal Process

    Semi-Markov Processes

    Markov Jump Processes

    Chapter 7: Markovian Arrival Processes


    Overview of Matrix-Analytic Methods

    Markovian Arrival Process

    Batch Markovian Arrival Process

    Markov-Modulated Poisson Process

    Markov-Modulated Bernoulli Process

    Sample Applications of MAP and Its Derivatives

    Chapter 8: Random Walk


    The Two-Dimensional Random Walk

    Random Walk as a Markov Chain

    Symmetric Random Walk as a Martingale

    Random Walk with Barriers

    Gambler’s Ruin

    First Return Times

    First Passage Times

    Maximum of a Random Walk

    Correlated Random Walk

    Continuous-time Random Walk

    Sample Applications of Random Walk

    Chapter 9: Brownian Motion and Diffusion Processes


    Brownian Motion

    Introduction to Stochastic Calculus

    Geometric Brownian Motion

    Fractional Brownian Motion

    Application of Brownian Motion to Option Pricing

    Random Walk Approximation of Brownian Motion

    The Ornstein-Uhlenbeck Process

    Diffusion Processes

    Examples of Diffusion Processes

    Relationship Between the Diffusion Process and Random Walk

    Chapter 10: Controlled Markov Processes


    Markov Decision Processes

    Semi-Markov Decision Processes

    Partially Observable Markov Decision Processes

    Chapter 11: Hidden Markov Models


    HMM Basics

    HMM Assumptions

    Three Fundamental Problems

    Solution Methods

    Types of Hidden Markov Models

    Hidden Markov Models with Silent States

    Extensions of Hidden Markov Models

    Other Extensions of HMM

    Chapter 12: Markov Point Processes

    Point Processes

    Temporal Point Processes

    Spatial Point Processes

    Spatial-Temporal Point Processes

    Operations on Point Processes

    Marked Point Processes

    Markov Point Processes

    Markov Marked Point Processes

    Applications of Markov Point Processes

Product details

  • No. of pages: 514
  • Language: English
  • Copyright: © Elsevier 2013
  • Published: May 22, 2013
  • Imprint: Elsevier
  • Paperback ISBN: 9780323282956
  • eBook ISBN: 9780124078390

About the Author

Oliver Ibe

Dr Ibe has been teaching at U Mass since 2003. He also has more than 20 years of experience in the corporate world, most recently as Chief Technology Officer at Sineria Networks and Director of Network Architecture for Spike Broadband Corp.

Affiliations and Expertise

University of Massachusetts, Lowell, USA

Ratings and Reviews

Write a review

There are currently no reviews for "Markov Processes for Stochastic Modeling"