Machine Learning and Data Science in the Oil and Gas Industry

Machine Learning and Data Science in the Oil and Gas Industry

Best Practices, Tools, and Case Studies

1st Edition - March 1, 2021

Write a review

  • Editor: Patrick Bangert
  • eBook ISBN: 9780128209141
  • Paperback ISBN: 9780128207147

Purchase options

Purchase options
DRM-free (EPub, PDF, Mobi)
Sales tax will be calculated at check-out

Institutional Subscription

Free Global Shipping
No minimum order


Machine Learning and Data Science in the Oil and Gas Industry explains how machine learning can be specifically tailored to oil and gas use cases. Petroleum engineers will learn when to use machine learning, how it is already used in oil and gas operations, and how to manage the data stream moving forward. Practical in its approach, the book explains all aspects of a data science or machine learning project, including the managerial parts of it that are so often the cause for failure. Several real-life case studies round out the book with topics such as predictive maintenance, soft sensing, and forecasting. Viewed as a guide book, this manual will lead a practitioner through the journey of a data science project in the oil and gas industry circumventing the pitfalls and articulating the business value.

Key Features

  • Chart an overview of the techniques and tools of machine learning including all the non-technological aspects necessary to be successful
  • Gain practical understanding of machine learning used in oil and gas operations through contributed case studies
  • Learn change management skills that will help gain confidence in pursuing the technology
  • Understand the workflow of a full-scale project and where machine learning benefits (and where it does not)


Oil and gas industry expert and practitioner working either in exploration, drilling, completions, engineering, production, maintenance or management

Table of Contents

  • 1. Introduction

     1.1 Who this book is for
     1.2 Preview of the content
     1.3 Oil and gas industry overview
     1.4 Brief history of oil exploration
     1.5 Oil and gas as limited resources
     1.6 Challenges of oil and gas

    2. Data Science, Statistics, and Time-Series

     2.1 Measurement, uncertainty, and record keeping
        2.1.1 Uncertainty
        2.1.2 Record keeping
     2.2 Correlation and timescales
     2.3 The idea of a model
     2.4 First principles models
     2.5 The straight line
     2.6 Representation and significance
     2.7 Outlier detection
     2.8 Residuals and statistical distributions
     2.9 Feature engineering
     2.10 Principal component analysis
     2.11 Practical advice

    3. Machine Learning
     3.1 Basic ideas of machine learning
     3.2 Bias-variance complexity trade-off
     3.3 Model types
        3.3.1 Deep neural network
        3.3.2 Recurrent neural network or long short-term memory network
        3.3.3 Support vector machines
        3.3.4 Random forest or decision trees
        3.3.5 Self-organizing maps (SOM)
        3.3.6 Bayesian network and ontology
     3.4 Training and assessing a model
     3.5 How good is my model?
     3.6 Role of domain knowledge
     3.7 Optimization using a model
     3.8 Practical advice

    4. Introduction to Machine Learning in the Oil and Gas Industry

     4.1 Forecasting
     4.2 Predictive maintenance
     4.3 Production
     4.4 Modeling physical relationships
     4.5 Optimization and advanced process control
     4.6 Other applications

    5. Data Management from the DCS to the Historian

     5.1 Introduction
        5.1.1 Convergence of OT and IT
        5.1.2 A maturity model for OT/IT convergence
        5.1.3 Digital Oilfield 2.0 headed to the edge
     5.2 Sensor data
        5.2.1. There are problems with data from sensors:
     data quality challenges
        5.2.2. Validation, estimation, and editing (VEE)
     5.3. Time series data
     5.4. How sensor data is transmitted by field networks
        5.4.1. From Plant to Field: Communications Protocols (HART, Fieldbus, OPC, OPC-UA and Wireless Hart)
        5.4.2. Wireless SCADA radio
        5.4.3. Which protocol is best?
     5.5. How control systems manage data
        5.5.1. Cloud-based SCADA and web-based SCADA
     5.6. Historians and information servers as a data source
        5.6.1. What can you record in a data historian?
     5.7. Data visualization of time series data—HMI (human machine interface)
        5.7.1. Asset performance management systems (APM)
        5.7.2. Key elements of data management for asset performance management
     5.8. Data management for equipment and facilities
        5.8.1. What is a document management system?
     5.9. Simulators, process modeling, and operating training systems
     5.10. How to get data out of the field/plant and to your analytics platform
        5.10.1. Data visualization
        5.10.2. Data analytics
        5.10.3. Three historical stages of industrial analytics
     5.11. Conclusion: do you know if your data is correct?

    6. Getting the Most Across the Value Chain

     6.1. Thinking outside the box
     6.2. Costing a project
     6.3. Valuing a project
        6.3.1. How to measure the benefit
        6.3.2. Measuring the benefit
     6.4. The business case
     6.5. Growing markets, optimizing networks
     6.6. Integrated strategy and alignment
     6.7. Case studies: capturing market opportunities
     6.8. Digital platform: partner, acquire, or build?
     6.9. What success looks like

    7. Project Management for a Machine Learning Project

     7.1. Classical project management in oil & gas—a (short) primer
     7.2. Agile—the mindset
     7.3. Scrum—the framework
        7.3.1. Roles of scrum
        7.3.2. Events
        7.3.3. Artifacts
        7.3.4. Values
        7.3.5. How it works
     7.4. Project execution—from pilot to product
        7.4.1. Pilot setup
        7.4.2. Product owner
        7.4.3. Development team
        7.4.4. Scrum master
        7.4.5. Stakeholders
     7.5. Management of change and culture
     7.6. Scaling—from pilot to product
        7.6.1. Take advantage of a platform
        7.6.2. Establish a team and involve the assets
        7.6.3. Keep developing
        7.6.4. Involve UX expertise

    8. The Business of AI Adoption

     8.1. Defining artificial intelligence
     8.2. AI impacts on oil and gas
        8.2.1. Upstream impacts
        8.2.2. Downstream impacts
        8.2.3. Production and midstream impacts
        8.2.4. New business models
     8.3. The adoption challenge
        8.3.1. The uncertainties of new technology
        8.3.2. AI in the field
     8.4. The problem of trust
        8.4.1 Work is evolving
        8.4.2. Driverless transportation
        8.4.3. Trust and the machine
        8.4.4. The human-smart machine trust gap
        8.4.5. Trusting a smart machine
        8.4.6. Trusting the smart machine developer
        8.4.7. Making it real
        8.4.8. Getting to trust
     8.5. Digital leaders lead
        8.5.1. Finding the digital leader
        8.5.2. Moving beyond trials and pilots
        8.5.3. The role of trials and pilots
        8.5.4. The economics of pilot projects
        8.5.5. Moving to enterprise deployment
     8.6. Overcoming barriers to scaling up
        8.6.1. The scale mismatch
        8.6.2. Supplier consolidation
        8.6.3. The corporate accelerator
        8.6.4. The oil company investor
     8.7. Confronting front line change
        8.7.1. The corporate parallels
        8.7.2. Early warning signs
     8.8. Doing digital change
        8.8.1. A typical change champion
        8.8.2. Organizational reaction to change

    9. Global Practice of AI and Big Data in Oil and Gas Industry

     9.1. Introduction
     9.2. Integrate digital rock physics with AI to optimize oil recovery
        9.2.1. The upstream business
        9.2.2. Digital core technology
        9.2.3. Modeling wettability at the pore-scale
     9.3. The molecular level advance planning system for refining
        9.3.1. Prediction of crude oil mixing and molecular properties
        9.3.2. Scheduling optimization at the molecular leve
        9.3.3. Collaborative optimization of the entire industry chain
     9.4. The application of big data in the oil refining process
        9.4.1. Principle and methodology
        9.4.2. A case study of CCR process unit
     9.5. Equipment management based on AI
        9.5.1. Equipment hazard monitoring and warning
        9.5.2. Equipment fault recognition and diagnosis
        9.5.3. Equipment health status, residual life prediction and other management

    10. Soft Sensors for NOx Emissions

     10.1. Introduction to soft sensing
     10.2. NOx and SOx emissions
     10.3. Combined heat and power (CHP)
     10.4. Soft sensing and machine learning
     10.5. Setting up a soft sensor
     10.6. Assessing the model
     10.7. Conclusion

    11. Detecting Electric Submersible Pump Failures
     11.1. Introduction
     11.2. ESP data analytic
     11.3. Principal Component Analysis
     11.4. PCA diagnostic model
     11.5. Case study: diagnosis of the ESP broken shaft
        11.5.1. Selection of the ESP broken shaft variables
        11.5.2. Score of principle components
        11.5.3. Pump broken shaft identification
     11.6. Conclusions

    12. Predictive and Diagnostic Maintenance for Rod Pumps

     12.1. Introduction
        12.1.1. Beam pumps
        12.1.2. Beam pump problems
        12.1.3. Problem statement
     12.2. Feature engineering
        12.2.1. Library-based methods
        12.2.2. Model-based methods
        12.2.3. Segment-based methods
        12.2.4. Other methods
        12.2.5. Selection of features
     12.3. Project method to validate our model
        12.3.1. Data collection
        12.3.2. Generation of training data
        12.3.3. Feature engineering
        12.3.4. Machine learning
        12.3.5. Summary of methodology
     12.4. Results
        12.4.1. Summary and review

    13. Forecasting Slugging in Gas Lift Wells

     13.1. Introduction
     13.2. Methodology
     13.3. Focus projects
        13.3.1. Dashboarding landscape/architecture
     13.3.2. Slugging
     13.4. Data structure
     13.5. Outlook
     13.6. Conclusion
     Further reading


Product details

  • No. of pages: 306
  • Language: English
  • Copyright: © Gulf Professional Publishing 2021
  • Published: March 1, 2021
  • Imprint: Gulf Professional Publishing
  • eBook ISBN: 9780128209141
  • Paperback ISBN: 9780128207147

About the Editor

Patrick Bangert

Dr. Patrick Bangert is the Vice President of Artificial Intelligence at Samsung SDS where he leads both the AI software development and AI consulting groups that each provide various offerings to the industry. He is the founder and Board Chair of Algorithmica Technologies, providing real-time process modeling, optimization, and predictive maintenance solutions to the process industry with a focus on chemistry and power generation. His doctorate from UCL specialized in applied mathematics, and his academic positions at NASA’s Jet Propulsion Laboratory and Los Alamos National Laboratory made use of optimization and machine learning for magnetohydrodynamics and particle accelerator experiments. He has published extensively across optimization and machine learning and their relevant applications in the real world.

Affiliations and Expertise

Vice President of Artificial Intelligence at Samsung SDSA, San Jose, CA, United States, and Founder and Board Chair of Algorithmica Technologies GmbH, Bad Nauheim, Germany

Ratings and Reviews

Write a review

There are currently no reviews for "Machine Learning and Data Science in the Oil and Gas Industry"