Logging & Monitoring

Introduction

Production needs centralized logs and metrics. Docker integrates with most observability stacks via log drivers and sidecar exporters.

Purpose of this lesson

This lesson teaches Logging & Monitoring as an engineering decision: what problem it solves, when to use it, how to implement it safely, and what signals tell you it is failing.

Understanding the topic

Use this in production readiness work. These practices decide whether containers are merely running or are actually secure, observable, recoverable, and safe to promote through a delivery pipeline.

Core concepts to understand:

• Log drivers: json-file (default), syslog, journald, gelf, fluentd, awslogs.
• Metrics: Prometheus + cAdvisor; Docker engine exposes its own endpoint.
• Traces: OpenTelemetry agents as sidecars or DaemonSets.

Visual explanation

Architecture or command flow to keep in mind:

# daemon.json — JSON log rotation
{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "5" }
}

# Or ship to Loki
docker run --log-driver=loki \
  --log-opt loki-url="http://loki:3100/loki/api/v1/push" myapp

Step-by-step explanation

1. Define the production risk first: credential leakage, vulnerable base image, runaway resource use, missing logs, or broken delivery flow.
2. Apply the smallest control that reduces the risk: non-root user, read-only filesystem, scan gate, secret mount, limit, or log driver.
3. Prove the control with commands such as docker inspect, docker history, docker stats, and a failing test case.
4. Automate the check in CI or operational runbooks so it does not depend on memory.
5. Review the result after a real incident or release and tighten the policy where evidence shows gaps.

Informative example

Use the example below as a working baseline, then verify the runtime behavior instead of assuming the command or file is correct.

# daemon.json — JSON log rotation
{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "5" }
}

# Or ship to Loki
docker run --log-driver=loki \
  --log-opt loki-url="http://loki:3100/loki/api/v1/push" myapp

A production-minded check usually includes docker ps, docker logs, docker inspect, and one validation from outside the container such as curl, a database connection, or a registry pull.

# Verification loop for Logging & Monitoring
docker ps -a
docker logs --tail=100 <container-name>
docker inspect <container-or-image-name>
docker system df

Real-world use

Most teams pair cAdvisor (per-container metrics) + Prometheus + Grafana + Loki for a complete open-source observability stack — all running in Docker.

Enterprise use cases

In a mature engineering organization, Logging & Monitoring is documented as a repeatable pattern with approved base images, ownership labels, CI checks, security expectations, rollback notes, and troubleshooting commands. The difference between a tutorial and production practice is that every container decision must be observable, reviewable, and reversible.

Best practices

• Always cap json-file log size.
• Stream logs to a central store in prod.
• Add /metrics endpoints to apps and scrape with Prometheus.

Debugging tips

• Inspect effective runtime settings with docker inspect; do not assume the Dockerfile or Compose file applied as expected.
• For resource incidents, compare docker stats, exit codes, OOMKilled state, and host memory pressure.
• For supply-chain issues, trace the exact image digest from build logs to registry to deployment.

Optimization strategies

• Treat image size, CVE count, non-root execution, and digest pinning as release quality signals.
• Set log rotation and resource limits before the first production incident, not after disk or memory exhaustion.
• Cache builds by registry or CI cache backend while still scanning the final pushed image.

Advanced interview questions

Hands-on exercise

Create a small lab for Logging & Monitoring: run the example, inspect the created Docker object, intentionally introduce one mistake, and record the command that reveals the failure. The goal is not just to make the happy path work; it is to build operational reflexes.

# Hands-on lab scaffold
mkdir -p docker-logging-monitoring-lab
cd docker-logging-monitoring-lab

# Add the Dockerfile, compose.yml, or command from this lesson.
# Then run one happy-path test and one broken-path test.
docker version
docker info
docker system df

Summary

Logging & Monitoring matters because Docker is not only a packaging tool; it is a runtime, build, networking, storage, and delivery workflow. Treat each lesson as a production habit: make it repeatable, inspectable, secure, and easy to debug.