Logging and Log Management

The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management

By

  • Anton Chuvakin, Ph.D., Stony Brook University, Stony Brook, NY., is a recognized security expert in the field of log management and PCI DSS compliance.
  • Kevin Schmidt, is a team lead and senior software developer at SecureWorks, Inc.
  • Chris Phillips, Christopher Phillips is a manager and senior software developer at Dell SecureWorks, Inc.

Logging and Log Management: The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management introduces information technology professionals to the basic concepts of logging and log management. It provides tools and techniques to analyze log data and detect malicious activity. The book consists of 22 chapters that cover the basics of log data; log data sources; log storage technologies; a case study on how syslog-ng is deployed in a real environment for log collection; covert logging; planning and preparing for the analysis log data; simple analysis techniques; and tools and techniques for reviewing logs for potential problems. The book also discusses statistical analysis; log data mining; visualizing log data; logging laws and logging mistakes; open source and commercial toolsets for log data collection and analysis; log management procedures; and attacks against logging systems. In addition, the book addresses logging for programmers; logging and compliance with regulations and policies; planning for log analysis system deployment; cloud logging; and the future of log standards, logging, and log analysis. This book was written for anyone interested in learning more about logging and log management. These include systems administrators, junior security engineers, application developers, and managers.
View full description

Audience

Computer Security staff and program managers; system, network, and application administrators; computer security incident response teams; and others who are responsible for performing duties related to computer security log management.

 

Book information

  • Published: November 2012
  • Imprint: SYNGRESS
  • ISBN: 978-1-59749-635-3

Reviews

"The authors provide a way to simplify the complex process of analyzing large quantities of varied logs. The log management and log analysis approaches they recommend are addressed in detail."--Reference and Research Book News, August 2013
"…Anton Chuvakin and his co-authors Kevin Schmidt and Christopher Phillips bring significant real-world experience to the reader and an important book on the topic....For those that want to find the gold in their logs…[it] is a great resource that shows how to maximize the gold that often lays hidden in your large stores of log data."--RSA Conference, December 2012




Table of Contents


Acknowledgments

About the Authors

About the Technical Editor

Foreword

Preface

Chapter 1 Logs, Trees, Forest: The Big Picture

Introduction

Log Data Basics

What Is Log Data?

How is Log Data Transmitted and Collected?

What is a Log Message?

The Logging Ecosystem

A Look at Things to Come

Logs Are Underrated

Logs Can Be Useful

Resource Management

Intrusion Detection

Troubleshooting

Forensics

Boring Audit, Fun Discovery

People, Process, Technology

Security Information and Event Management (SIEM)

Summary

Chapter 2 What is a Log?

Introduction

Definitions

Logs? What logs?

Log Formats and Types

Log Syntax

Log Content

Criteria of Good Logging

Ideal Logging Scenario

Summary

Chapter 3 Log Data Sources

Introduction

Logging Sources

Syslog

SNMP

The Windows Event Log

Log Source Classification

Security-Related Host Logs

Security-Related Network Logs

Security Host Logs

Summary

Chapter 4 Log Storage Technologies

Introduction

Log Retention Policy

Log Storage Formats

Text-Based Log Files

Binary Files

Compressed Files

Database Storage of Log Data

Advantages

Disadvantages

Defining Database Storage Goals

Hadoop Log Storage

Advantages

Disadvantages

The Cloud and Hadoop

Getting Started with Amazon Elastic MapReduce

Navigating the Amazon

Uploading Logs to Amazon Simple Storage Services (S3)

Create a Pig Script to Analyze an Apache Access Log

Processing Log Data in Amazon Elastic MapReduce (EMR)

Log Data Retrieval and Archiving

Online

?Near-line

Offline

Summary

Chapter 5 syslog-ng Case Study

Introduction

Obtaining syslog-ng

What Is syslog-ngsyslog-ng?

Example Deployment

Configurations

Troubleshooting syslog-ng

Summary

Chapter 6 Covert Logging

Introduction

Complete Stealthy Log Setup

Stealthy Log Generation

Stealthy Pickup of Logs

IDS Log Source

Log Collection Server

“Fake” Server or Honeypot

Logging in Honeypots

Honeynet’s Shell Covert Keystroke Logger

Honeynet’s Sebek2 Case Study

Covert Channels for Logging Brief

Summary

Chapter 7 Analysis Goals, Planning, and Preparation: What Are We Looking For?

Introduction

Goals

Past Bad Things

Future Bad Things, Never Before Seen Things, and All But the Known Good Things

Planning

Accuracy

Integrity

Confidence

Preservation

Sanitization

Normalization

Challenges with Time

?Preparation

Separating Log Messages

Parsing

Data Reduction

Summary

Chapter 8 Simple Analysis Techniques

Introduction

Line by Line: Road to Despair

Simple Log Viewers

Real-Time Review

Historical Log Review

Simple Log Manipulation

Limitations of Manual Log Review

Responding to the Results of Analysis

Acting on Critical Logs

Acting on Summaries of Non-Critical Logs

Developing an Action Plan

Automated Actions

Examples

Incident Response Scenario

Routine Log Review

Summary

Chapter 9 Filtering, Normalization, and Correlation

Introduction

Filtering

Artificial Ignorance

Normalization

IP Address Validation

Snort

Windows Snare

Generic Cisco IOS Messages

Regular Expression Performance Concerns

Correlation

Micro-Level Correlation

Macro-Level Correlation

Using Data in Your Environment

Simple Event Correlator (SEC)

Stateful Rule Example

Building Your Own Rules Engine

Common Patterns to Look For

The Future

Summary

Chapter 10 Statistical Analysis

Introduction

Frequency

Baseline

Thresholds

Anomaly Detection

Windowing

Machine Learning

k-Nearest Neighbor (kNN)

Applying the k-NN Algorithm to Logs

Combining Statistical Analysis with Rules-Based Correlation

Summary

Chapter 11 Log Data Mining

Introduction

Data Mining Intro

Log Mining Intro

Log Mining Requirements

What We Mine For?

Deeper into Interesting

Summary

Chapter 12 Reporting and Summarization

Introduction

Defining the Best Reports

Authentication and Authorization Reports

Network Activity Reports

Why They Are Important

Specifics Reports

Who Can Use These Reports

Resource Access Reports

Why They Are Important

Specifics Reports

Who Can Use These Reports

Malware Activity Reports

Why They Are Important

Specific Reports

Who Can Use These Reports

Critical Errors and Failures Reports

Why They Are Important

Specifics Reports

Who Can Use These Reports

Summary

Chapter 13 Visualizing Log Data

Introduction

Visual Correlation

Real-Time Visualization

Treemaps

Log Data Constellations

Traditional Log Data Graphing

Summary

Chapter 14 Logging Laws and Logging Mistakes

Introduction

Logging Laws

Law 1-Law of Collection

Law 2-Law of Retention

Law 3-Law of Monitoring

Law 3-Law of Availability

Law 4-Law of Security

Law 5-Law of Constant Changes

Logging Mistakes

Not Logging at All

Not Looking at Log Data

Storing for Too Short a Time

Prioritizing Before Collection

Ignoring Application Logs

Only Looking for Known Bad Entries

Summary

Chapter 15 Tools for Log Analysis and Collection

Introduction

Outsource, Build, or Buy

Building a Solution

Buy

Outsource

Questions for You, Your Organization, and Vendors

Basic Tools for Log Analysis

Grep

Awk

Microsoft Log Parser

Other Basic Tools to Consider

The Role of the Basic Tools in Log Analysis

Utilities for Centralizing Log Information

Syslog

Rsyslog

Snare

Log Analysis Tools-Beyond the Basics

OSSEC

OSSIM

Other Analysis Tools to Consider

Commercial Vendors

Splunk

NetIQ Sentinel

IBM q1Labs

Loggly

Summary

Chapter 16 Log Management Procedures: Log Review, Response, and Escalation

Introduction

Assumptions, Requirements, and Precautions

Requirements

Precautions

Common Roles and Responsibilities

PCI and Log Data

Key Requirement 10

Other Requirements Related to Logging

Logging Policy

Review, Response, and Escalation Procedures and Workflows

Periodic Log Review Practices and Patterns

Building an Initial Baseline Using a Log Management Tool

Building an Initial Baseline Manually

Main Workflow: Daily Log Review

Exception Investigation and Analysis

Incident Response and Escalation

Validation of Log Review

Proof of Logging

Proof of Log Review

Proof of Exception Handling

Logbook-Evidence of Exception of Investigations

Recommended Logbook Format

Example Logbook Entry

PCI Compliance Evidence Package

Management Reporting

Periodic Operational Tasks

Daily Tasks

Weekly Tasks

Monthly Tasks

Quarterly Tasks

Annual Tasks

Additional Resources

Summary

Chapter 17 Attacks Against Logging Systems

Introduction

Attacks

What to Attack?

Attacks on Confidentiality

Attacks on Integrity

Attacks on Availability

Summary

Chapter 18 Logging for Programmers

Introduction

Roles and Responsibilities

Logging for Programmers

What Should Be Logged?

Logging APIs for Programmers

Log Rotation

Bad Log Messages

Log Message Formatting

Security Considerations

Performance Considerations

Summary

Chapter 19 Logs and Compliance

Introduction

PCI DSS

Key Requirement 10

ISO2700x Series

HIPAA

FISMA

NIST 800-53 Logging Guidance

Summary

Chapter 20 Planning Your Own Log Analysis System

Introduction

Planning

Roles and Responsibilities

Resources

Goals

Selecting Systems and Devices for Logging

Software Selection

Open Source

Commercial

Policy Definition

Logging Policy

Log File Rotation

Log Data Collection

Retention/Storage

Response

Architecture

Basic

Log Server and Log Collector

Log Server and Log Collector with Long-Term Storage

Distributed

Scaling

Summary

Chapter 21 Cloud Logging

Introduction

Cloud Computing

Service Delivery Models

Cloud Deployment Models

Characteristics of a Cloud Infrastructure

Standards? We Don’t Need No Stinking Standards!

Cloud Logging

A Quick Example: Loggly

Regulatory, Compliance, and Security Issues

Big Data in the Cloud

A Quick Example: Hadoop

SIEM in the Cloud

Pros and Cons of Cloud Logging

Cloud Logging Provider Inventory

Additional Resources

Summary

Chapter 22 Log Standards and Future Trends

Introduction

Extrapolations of Today to the Future

More Log Data

More Motivations

More Analysis

Log Future and Standards

Adoption Trends

Desired Future

Summary

Index