COVID-19 Update: We are currently shipping orders daily. However, due to transit disruptions in some geographies, deliveries may be delayed. To provide all customers with timely access to content, we are offering 50% off Science and Technology Print & eBook bundle options. Terms & conditions.
Markov Processes for Stochastic Modeling - 2nd Edition - ISBN: 9780124077959, 9780124078390

Markov Processes for Stochastic Modeling

2nd Edition

Author: Oliver Ibe
Hardcover ISBN: 9780124077959
Paperback ISBN: 9780323282956
eBook ISBN: 9780124078390
Imprint: Elsevier
Published Date: 22nd May 2013
Page Count: 514
Sales tax will be calculated at check-out Price includes VAT/GST
Price includes VAT/GST

Institutional Subscription

Secure Checkout

Personal information is secured with SSL technology.

Free Shipping

Free global shipping
No minimum order.


Markov processes are processes that have limited memory. In particular, their dependence on the past is only through the previous state. They are used to model the behavior of many systems including communications systems, transportation networks, image segmentation and analysis, biological systems and DNA sequence analysis, random atomic motion and diffusion in physics, social mobility, population studies, epidemiology, animal and insect migration, queueing systems, resource management, dams, financial engineering, actuarial science, and decision systems.

Covering a wide range of areas of application of Markov processes, this second edition is revised to highlight the most important aspects as well as the most recent trends and applications of Markov processes. The author spent over 16 years in the industry before returning to academia, and he has applied many of the principles covered in this book in multiple research projects. Therefore, this is an applications-oriented book that also includes enough theory to provide a solid ground in the subject for the reader.

Key Features

  • Presents both the theory and applications of the different aspects of Markov processes
  • Includes numerous solved examples as well as detailed diagrams that make it easier to understand the principle being presented
  • Discusses different applications of hidden Markov models, such as DNA sequence analysis and speech analysis.


Graduate and upper-level undergraduate students, researchers and practitioners working in Markov Processes.

Table of Contents

Chapter 1: Basic Concepts

Review of Probability

Random Variables

Transform Methods

Bivariate Random Variables

Many Random Variables

Fubini’s Theorem

Sums of Independent Random Variables

Some Probability Distributions

Introduction to Stochastic Processes

Classification of Stochastic Processes

Characterizing a Stochastic Process

Stationary Stochastic Processes

Ergodic Stochastic Processes

Some Models of Stochastic Processes

Chapter 2: Introduction to Markov Processes


Structure of Markov Processes

Strong Markov Property

Applications of Discrete-time Markov Processes

Applications of Continuous-time Markov Processes

Applications of Continuous-state Markov Processes

Chapter 3: Discrete-Time Markov Chains


State Transition Probability Matrix

State Transition Diagrams

Classification of States

Limiting-State Probabilities

Sojourn Time

Transient Analysis of Discrete-Time Markov Chains

First Passage and Recurrence Times

Occupancy Times

Absorbing Markov Chains and the Fundamental Matrix

Reversible Markov Chains

Chapter 4: Continuous-Time Markov Chains


Transient Analysis

Birth and Death Processes

First Passage Time

The Uniformization Method

Reversible Continuous-Time Markov Chains

Chapter 5: Markovian Queueing Systems


Description of a Queueing System

The Kendall Notation

The Little’s Formula

The PASTA Property

The M/M/1 Queueing System

Examples of Other M/M Queueing Systems

M/G/1 Queue

G/M/1 Queue

Chapter 6: Markov Renewal Processes

Renewal Processes

The Renewal Equation

The Elementary Renewal Theorem

Random Incidence and Residual Time

Markov Renewal Process

Semi-Markov Processes

Markov Jump Processes

Chapter 7: Markovian Arrival Processes


Overview of Matrix-Analytic Methods

Markovian Arrival Process

Batch Markovian Arrival Process

Markov-Modulated Poisson Process

Markov-Modulated Bernoulli Process

Sample Applications of MAP and Its Derivatives

Chapter 8: Random Walk


The Two-Dimensional Random Walk

Random Walk as a Markov Chain

Symmetric Random Walk as a Martingale

Random Walk with Barriers

Gambler’s Ruin

First Return Times

First Passage Times

Maximum of a Random Walk

Correlated Random Walk

Continuous-time Random Walk

Sample Applications of Random Walk

Chapter 9: Brownian Motion and Diffusion Processes


Brownian Motion

Introduction to Stochastic Calculus

Geometric Brownian Motion

Fractional Brownian Motion

Application of Brownian Motion to Option Pricing

Random Walk Approximation of Brownian Motion

The Ornstein-Uhlenbeck Process

Diffusion Processes

Examples of Diffusion Processes

Relationship Between the Diffusion Process and Random Walk

Chapter 10: Controlled Markov Processes


Markov Decision Processes

Semi-Markov Decision Processes

Partially Observable Markov Decision Processes

Chapter 11: Hidden Markov Models


HMM Basics

HMM Assumptions

Three Fundamental Problems

Solution Methods

Types of Hidden Markov Models

Hidden Markov Models with Silent States

Extensions of Hidden Markov Models

Other Extensions of HMM

Chapter 12: Markov Point Processes

Point Processes

Temporal Point Processes

Spatial Point Processes

Spatial-Temporal Point Processes

Operations on Point Processes

Marked Point Processes

Markov Point Processes

Markov Marked Point Processes

Applications of Markov Point Processes


No. of pages:
© Elsevier 2013
22nd May 2013
Hardcover ISBN:
Paperback ISBN:
eBook ISBN:

About the Author

Oliver Ibe

Dr Ibe has been teaching at U Mass since 2003. He also has more than 20 years of experience in the corporate world, most recently as Chief Technology Officer at Sineria Networks and Director of Network Architecture for Spike Broadband Corp.

Affiliations and Expertise

University of Massachusetts, Lowell, USA


"Markov processes are the most popular modeling tools for stochastic systems in many different fields, and Ibe compiles in a single volume many of the Markovian models used indifferent disciplines. The information could be useful to graduate students and researchers in any field that uses Markov processes, he says, but he was thinking particularly of those in traffic engineering, image analysis, bioinformatics, biostatistics, financial engineering, and computational biology."--Reference & Research Book News, October 2013

Ratings and Reviews