Logging and Log Management book cover

Logging and Log Management

The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management

Logging and Log Management: The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management introduces information technology professionals to the basic concepts of logging and log management. It provides tools and techniques to analyze log data and detect malicious activity. The book consists of 22 chapters that cover the basics of log data; log data sources; log storage technologies; a case study on how syslog-ng is deployed in a real environment for log collection; covert logging; planning and preparing for the analysis log data; simple analysis techniques; and tools and techniques for reviewing logs for potential problems. The book also discusses statistical analysis; log data mining; visualizing log data; logging laws and logging mistakes; open source and commercial toolsets for log data collection and analysis; log management procedures; and attacks against logging systems. In addition, the book addresses logging for programmers; logging and compliance with regulations and policies; planning for log analysis system deployment; cloud logging; and the future of log standards, logging, and log analysis. This book was written for anyone interested in learning more about logging and log management. These include systems administrators, junior security engineers, application developers, and managers.

Audience

Computer Security staff and program managers; system, network, and application administrators; computer security incident response teams; and others who are responsible for performing duties related to computer security log management.

Paperback, 460 Pages

Published: November 2012

Imprint: Syngress

ISBN: 978-1-59749-635-3

Reviews

  • "The authors provide a way to simplify the complex process of analyzing large quantities of varied logs. The log management and log analysis approaches they recommend are addressed in detail."--Reference and Research Book News, August 2013
    "…Anton Chuvakin and his co-authors Kevin Schmidt and Christopher Phillips bring significant real-world experience to the reader and an important book on the topic....For those that want to find the gold in their logs…[it] is a great resource that shows how to maximize the gold that often lays hidden in your large stores of log data."--RSA Conference, December 2012


Contents


  • Acknowledgments

    About the Authors

    About the Technical Editor

    Foreword

    Preface

    Chapter 1 Logs, Trees, Forest: The Big Picture

    Introduction

    Log Data Basics

    What Is Log Data?

    How is Log Data Transmitted and Collected?

    What is a Log Message?

    The Logging Ecosystem

    A Look at Things to Come

    Logs Are Underrated

    Logs Can Be Useful

    Resource Management

    Intrusion Detection

    Troubleshooting

    Forensics

    Boring Audit, Fun Discovery

    People, Process, Technology

    Security Information and Event Management (SIEM)

    Summary

    Chapter 2 What is a Log?

    Introduction

    Definitions

    Logs? What logs?

    Log Formats and Types

    Log Syntax

    Log Content

    Criteria of Good Logging

    Ideal Logging Scenario

    Summary

    Chapter 3 Log Data Sources

    Introduction

    Logging Sources

    Syslog

    SNMP

    The Windows Event Log

    Log Source Classification

    Security-Related Host Logs

    Security-Related Network Logs

    Security Host Logs

    Summary

    Chapter 4 Log Storage Technologies

    Introduction

    Log Retention Policy

    Log Storage Formats

    Text-Based Log Files

    Binary Files

    Compressed Files

    Database Storage of Log Data

    Advantages

    Disadvantages

    Defining Database Storage Goals

    Hadoop Log Storage

    Advantages

    Disadvantages

    The Cloud and Hadoop

    Getting Started with Amazon Elastic MapReduce

    Navigating the Amazon

    Uploading Logs to Amazon Simple Storage Services (S3)

    Create a Pig Script to Analyze an Apache Access Log

    Processing Log Data in Amazon Elastic MapReduce (EMR)

    Log Data Retrieval and Archiving

    Online

    ?Near-line

    Offline

    Summary

    Chapter 5 syslog-ng Case Study

    Introduction

    Obtaining syslog-ng

    What Is syslog-ngsyslog-ng?

    Example Deployment

    Configurations

    Troubleshooting syslog-ng

    Summary

    Chapter 6 Covert Logging

    Introduction

    Complete Stealthy Log Setup

    Stealthy Log Generation

    Stealthy Pickup of Logs

    IDS Log Source

    Log Collection Server

    “Fake” Server or Honeypot

    Logging in Honeypots

    Honeynet’s Shell Covert Keystroke Logger

    Honeynet’s Sebek2 Case Study

    Covert Channels for Logging Brief

    Summary

    Chapter 7 Analysis Goals, Planning, and Preparation: What Are We Looking For?

    Introduction

    Goals

    Past Bad Things

    Future Bad Things, Never Before Seen Things, and All But the Known Good Things

    Planning

    Accuracy

    Integrity

    Confidence

    Preservation

    Sanitization

    Normalization

    Challenges with Time

    ?Preparation

    Separating Log Messages

    Parsing

    Data Reduction

    Summary

    Chapter 8 Simple Analysis Techniques

    Introduction

    Line by Line: Road to Despair

    Simple Log Viewers

    Real-Time Review

    Historical Log Review

    Simple Log Manipulation

    Limitations of Manual Log Review

    Responding to the Results of Analysis

    Acting on Critical Logs

    Acting on Summaries of Non-Critical Logs

    Developing an Action Plan

    Automated Actions

    Examples

    Incident Response Scenario

    Routine Log Review

    Summary

    Chapter 9 Filtering, Normalization, and Correlation

    Introduction

    Filtering

    Artificial Ignorance

    Normalization

    IP Address Validation

    Snort

    Windows Snare

    Generic Cisco IOS Messages

    Regular Expression Performance Concerns

    Correlation

    Micro-Level Correlation

    Macro-Level Correlation

    Using Data in Your Environment

    Simple Event Correlator (SEC)

    Stateful Rule Example

    Building Your Own Rules Engine

    Common Patterns to Look For

    The Future

    Summary

    Chapter 10 Statistical Analysis

    Introduction

    Frequency

    Baseline

    Thresholds

    Anomaly Detection

    Windowing

    Machine Learning

    k-Nearest Neighbor (kNN)

    Applying the k-NN Algorithm to Logs

    Combining Statistical Analysis with Rules-Based Correlation

    Summary

    Chapter 11 Log Data Mining

    Introduction

    Data Mining Intro

    Log Mining Intro

    Log Mining Requirements

    What We Mine For?

    Deeper into Interesting

    Summary

    Chapter 12 Reporting and Summarization

    Introduction

    Defining the Best Reports

    Authentication and Authorization Reports

    Network Activity Reports

    Why They Are Important

    Specifics Reports

    Who Can Use These Reports

    Resource Access Reports

    Why They Are Important

    Specifics Reports

    Who Can Use These Reports

    Malware Activity Reports

    Why They Are Important

    Specific Reports

    Who Can Use These Reports

    Critical Errors and Failures Reports

    Why They Are Important

    Specifics Reports

    Who Can Use These Reports

    Summary

    Chapter 13 Visualizing Log Data

    Introduction

    Visual Correlation

    Real-Time Visualization

    Treemaps

    Log Data Constellations

    Traditional Log Data Graphing

    Summary

    Chapter 14 Logging Laws and Logging Mistakes

    Introduction

    Logging Laws

    Law 1-Law of Collection

    Law 2-Law of Retention

    Law 3-Law of Monitoring

    Law 3-Law of Availability

    Law 4-Law of Security

    Law 5-Law of Constant Changes

    Logging Mistakes

    Not Logging at All

    Not Looking at Log Data

    Storing for Too Short a Time

    Prioritizing Before Collection

    Ignoring Application Logs

    Only Looking for Known Bad Entries

    Summary

    Chapter 15 Tools for Log Analysis and Collection

    Introduction

    Outsource, Build, or Buy

    Building a Solution

    Buy

    Outsource

    Questions for You, Your Organization, and Vendors

    Basic Tools for Log Analysis

    Grep

    Awk

    Microsoft Log Parser

    Other Basic Tools to Consider

    The Role of the Basic Tools in Log Analysis

    Utilities for Centralizing Log Information

    Syslog

    Rsyslog

    Snare

    Log Analysis Tools-Beyond the Basics

    OSSEC

    OSSIM

    Other Analysis Tools to Consider

    Commercial Vendors

    Splunk

    NetIQ Sentinel

    IBM q1Labs

    Loggly

    Summary

    Chapter 16 Log Management Procedures: Log Review, Response, and Escalation

    Introduction

    Assumptions, Requirements, and Precautions

    Requirements

    Precautions

    Common Roles and Responsibilities

    PCI and Log Data

    Key Requirement 10

    Other Requirements Related to Logging

    Logging Policy

    Review, Response, and Escalation Procedures and Workflows

    Periodic Log Review Practices and Patterns

    Building an Initial Baseline Using a Log Management Tool

    Building an Initial Baseline Manually

    Main Workflow: Daily Log Review

    Exception Investigation and Analysis

    Incident Response and Escalation

    Validation of Log Review

    Proof of Logging

    Proof of Log Review

    Proof of Exception Handling

    Logbook-Evidence of Exception of Investigations

    Recommended Logbook Format

    Example Logbook Entry

    PCI Compliance Evidence Package

    Management Reporting

    Periodic Operational Tasks

    Daily Tasks

    Weekly Tasks

    Monthly Tasks

    Quarterly Tasks

    Annual Tasks

    Additional Resources

    Summary

    Chapter 17 Attacks Against Logging Systems

    Introduction

    Attacks

    What to Attack?

    Attacks on Confidentiality

    Attacks on Integrity

    Attacks on Availability

    Summary

    Chapter 18 Logging for Programmers

    Introduction

    Roles and Responsibilities

    Logging for Programmers

    What Should Be Logged?

    Logging APIs for Programmers

    Log Rotation

    Bad Log Messages

    Log Message Formatting

    Security Considerations

    Performance Considerations

    Summary

    Chapter 19 Logs and Compliance

    Introduction

    PCI DSS

    Key Requirement 10

    ISO2700x Series

    HIPAA

    FISMA

    NIST 800-53 Logging Guidance

    Summary

    Chapter 20 Planning Your Own Log Analysis System

    Introduction

    Planning

    Roles and Responsibilities

    Resources

    Goals

    Selecting Systems and Devices for Logging

    Software Selection

    Open Source

    Commercial

    Policy Definition

    Logging Policy

    Log File Rotation

    Log Data Collection

    Retention/Storage

    Response

    Architecture

    Basic

    Log Server and Log Collector

    Log Server and Log Collector with Long-Term Storage

    Distributed

    Scaling

    Summary

    Chapter 21 Cloud Logging

    Introduction

    Cloud Computing

    Service Delivery Models

    Cloud Deployment Models

    Characteristics of a Cloud Infrastructure

    Standards? We Don’t Need No Stinking Standards!

    Cloud Logging

    A Quick Example: Loggly

    Regulatory, Compliance, and Security Issues

    Big Data in the Cloud

    A Quick Example: Hadoop

    SIEM in the Cloud

    Pros and Cons of Cloud Logging

    Cloud Logging Provider Inventory

    Additional Resources

    Summary

    Chapter 22 Log Standards and Future Trends

    Introduction

    Extrapolations of Today to the Future

    More Log Data

    More Motivations

    More Analysis

    Log Future and Standards

    Adoption Trends

    Desired Future

    Summary

    Index






Advertisement

advert image