Logging Systems

Introduction

Logging systems store huge volumes of append-only, time-ordered events with read patterns dominated by recent + filtered queries. MongoDB time-series collections, capped collections and TTL indexes were built for this exact shape — many platforms use MongoDB as the primary log store before (or instead of) Elasticsearch.

The trade-off: MongoDB shines for structured application logs (audit, business events, traces) where you want rich JSON, GraphQL-style filtering and aggregations. For full-text log search at TB scale, pair it with Atlas Search or stream to Elastic.

Understanding the topic

Designing a log store on MongoDB:

• Time-series collection with metaField: { service, env } — dense bucketing per service.
• TTL index on ts to auto-expire after N days (7d hot, archive cold).
• Structured fields — level, traceId, userId, msg, attrs.
• Compound index on (service, level, ts desc) for the standard "errors in service X last hour" query.
• Atlas Search for full-text on msg and attrs.*.
• Online Archive tiers cold logs to S3 — pay cents per GB while keeping query access.

Informative example

Production log ingest + query:

db.createCollection("logs", {
  timeseries: { timeField: "ts", metaField: "ctx", granularity: "seconds" },
  expireAfterSeconds: 7 * 24 * 3600
});
db.logs.createIndex({ "ctx.service": 1, level: 1, ts: -1 });

// Bulk insert from your log shipper (Fluent Bit, Vector, OpenTelemetry)
db.logs.insertMany(batch.map(e => ({
  ts: new Date(e.ts),
  ctx: { service: e.service, env: e.env, host: e.host },
  level: e.level, traceId: e.traceId, userId: e.userId,
  msg: e.msg, attrs: e.attrs
})));

// Tail errors for one service
db.logs.find({ "ctx.service": "checkout", level: "ERROR", ts: { $gte: lastHour } })
       .sort({ ts: -1 }).limit(200);

Real-world use

Many internal platforms at fintech and SaaS companies store application logs in MongoDB Atlas time-series with 7-day hot retention and 90-day Online Archive — total cost a fraction of running self-hosted Elasticsearch.

Best practices

• Always batch inserts (insertMany) from log shippers — never one document per call.
• Use a TTL index to enforce retention; never rely on a delete cron.
• Add Atlas Search only on fields you actually full-text search — costs scale with index size.
• Sample DEBUG/TRACE logs in production — only INFO+ goes to durable storage.

Common mistakes

• One document per log line in a regular collection — explodes index and oplog cost.
• Storing huge stack traces in every log — strip or truncate; archive raw to S3.
• No retention strategy — clusters fill up and writes start failing.