Log Aggregation - Memory Buffer Caused Silent 20-Minute Gap

Production systems are lying to you right now — not maliciously, but by omission. Every microservice, container, and serverless function writes its own story to its own local log file. The moment something breaks at 2 a.m., that story is scattered across dozens of machines that may not even exist by morning. Logs that live only on the box they were generated on are worse than useless — they're a false sense of security.

Log aggregation solves one problem: get every event from every component into one place, consistently, fast enough to act on. Without it, you're debugging in the dark. With it, you can trace a single user's failed checkout across a frontend service, auth service, payments API, and database — in seconds, not hours. The difference between a 5-minute MTTR and a 5-hour one is almost always a well-designed logging pipeline.

This guide covers structured logging, disk-backed buffers, tiered retention, and the three mistakes that silently kill observability. These are patterns pulled from real production environments — the kind that handle millions of events per day.

Why Log Aggregation Best Practices Are Not Optional

Log aggregation is the practice of centralizing logs from distributed services into a single, queryable platform — but the real mechanic is buffering. Without a buffer, every log write is a synchronous network call, which kills throughput under load. A memory buffer absorbs bursts, batches writes, and decouples your application from the logging backend. The trade-off: if the buffer flushes asynchronously, you can lose data on crash or, worse, silently delay delivery.

In practice, aggregation pipelines use a fixed-size in-memory queue (e.g., 10,000 events) that flushes every 5 seconds or when full. This gives O(1) enqueue and amortized O(n) flush cost. The critical property is backpressure: when the buffer is full, the logger must either block the calling thread (safe but slow) or drop events (fast but silent data loss). Most libraries default to dropping — that’s where the gap comes from.

Use memory buffering when latency matters more than perfect delivery — which is almost always. But you must configure a circuit breaker: if the remote endpoint is down for more than 30 seconds, switch to a disk-backed fallback. Without that, a brief network partition causes the buffer to fill, drop logs, and create a blackout window that looks like your app stopped working.

Structured Logging: Stop Writing Sentences, Start Writing Data

The single highest-leverage change you can make to your logging strategy costs zero dollars and takes one afternoon: switch from unstructured to structured logs.

Unstructured logs are prose. They look like this: ERROR: Payment failed for user 4821 after 3 retries at 14:32:01. A human can read it. A machine cannot reliably parse it. The moment you want to query 'show me all payment failures where retry_count > 2 in the last hour', you're writing fragile regex against free-form text. That breaks the moment someone changes the wording of the message.

Structured logs are data. Every log line is a JSON object (or logfmt key-value pairs) with consistent, queryable fields. The same event becomes: {"level":"error","event":"payment_failed","user_id":4821,"retry_count":3,"timestamp":"2024-01-15T14:32:01Z"}. Now your log aggregator can index retry_count as a number, and your query is a trivial filter — no regex, no fragility.

The discipline here is schema consistency. Define your fields organisation-wide: service_name, trace_id, user_id, duration_ms, level. Every team uses the same names. The payoff comes when you correlate events across services — and that only works if field names match.

A hard-won lesson: never log raw request bodies or response payloads. They contain PII, tokens, and credit card numbers. Log derived metadata instead: request_size_bytes, response_status, token_prefix. Your future self during a security audit will thank you.

One more pattern: use log sampling in hot paths. If a high-throughput endpoint logs on every request, your storage costs explode and your pipeline backs up. Use a counter: log the first occurrence, then every 100th. Keep errors always unsampled. This keeps your pipeline stable under burst traffic while still surfacing anomalies.

Schema versioning is another consideration. When you add or remove fields, older and newer log lines will coexist. Document your schema with version numbers. Plan for queries that span versions. A simple approach: include a 'log_schema_version' field. Start at 1. When you add a mandatory field, bump it. Aggregators can use this field to apply different parsing at query time.

import json
import logging
import time
import uuid
from datetime import datetime, timezone

class JsonFormatter(logging.Formatter):
    def __init__(self, service_name: str):
        super().__init__()
        self.service_name = service_name

    def format(self, record: logging.LogRecord) -> str:
        log_payload = {
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "level": record.levelname.lower(),
            "service": self.service_name,
            "message": record.getMessage(),
            "logger": record.name,
            "line": f"{record.filename}:{record.lineno}"
        }
        if hasattr(record, 'extra_fields'):
            log_payload.update(record.extra_fields)
        if record.exc_info:
            log_payload['exception'] = self.formatException(record.exc_info)
        return json.dumps(log_payload)

def create_logger(service_name: str) -> logging.Logger:
    logger = logging.getLogger(service_name)
    logger.setLevel(logging.DEBUG)
    handler = logging.StreamHandler()
    handler.setFormatter(JsonFormatter(service_name))
    logger.addHandler(handler)
    return logger

class ContextLogger:
    def __init__(self, logger: logging.Logger, context: dict):
        self._logger = logger
        self._context = context

    def info(self, message: str, **extra):
        self._log(logging.INFO, message, extra)

    def error(self, message: str, exc_info: bool = False, **extra):
        self._log(logging.ERROR, message, extra, exc_info)

    def _log(self, level: int, message: str, extra: dict, exc_info: bool = False):
        merged = {**self._context, **extra}
        record = self._logger.makeRecord(
            self._logger.name, level, "", 0, message, [], None
        )
        record.extra_fields = merged
        if exc_info:
            import sys
            record.exc_info = sys.exc_info()
        self._logger.handle(record)

def process_payment(order_id: str, user_id: int, amount_cents: int):
    base_logger = create_logger("payments-service")
    trace_id = str(uuid.uuid4())
    log = ContextLogger(base_logger, {
        "trace_id": trace_id,
        "order_id": order_id,
        "user_id": user_id,
    })
    log.info("payment_processing_started", amount_cents=amount_cents)
    start_time = time.monotonic()
    try:
        time.sleep(0.042)
        if amount_cents > 100_000:
            raise ValueError("Amount exceeds single-transaction limit")
        duration_ms = round((time.monotonic() - start_time) * 1000, 2)
        log.info("payment_succeeded",
                 amount_cents=amount_cents,
                 duration_ms=duration_ms,
                 gateway="stripe")
    except ValueError as exc:
        duration_ms = round((time.monotonic() - start_time) * 1000, 2)
        log.error("payment_failed",
                  exc_info=True,
                  amount_cents=amount_cents,
                  duration_ms=duration_ms,
                  failure_reason="limit_exceeded")
        raise

if __name__ == "__main__":
    print("--- Successful payment ---")
    process_payment(order_id="ORD-9921", user_id=4821, amount_cents=4999)
    print("\n--- Failed payment ---")
    try:
        process_payment(order_id="ORD-9922", user_id=4821, amount_cents=150_000)
    except ValueError:
        pass

Building a Pipeline That Doesn't Lose Messages Under Load

Getting logs off the machine that produced them is harder than it sounds. Most teams get this wrong in one of two ways: they either block the application while waiting for log writes to complete, or they drop messages silently when the downstream system is slow. Both failures cost you exactly when you need observability the most — during an incident.

The canonical architecture for a production log pipeline is: Application → Local Agent → Message Buffer → Aggregator → Storage. Each arrow is an asynchronous boundary. The application never waits for a log to reach Elasticsearch. It writes to stdout. A sidecar agent (Fluent Bit, Filebeat) tails that output and ships it forward. A buffer (disk-backed in the agent, or an external queue like Kafka for very high volumes) absorbs spikes. The aggregator (Logstash, Fluentd) processes and routes. Storage (Elasticsearch, Loki, CloudWatch) persists.

The local agent is your reliability contract. Configure it with a disk-backed buffer so if the aggregator goes down for 10 minutes, the agent stores messages locally and replays them when connectivity restores. Without this, a 10-minute aggregator restart means a 10-minute gap in your logs — right when you're trying to understand what caused the aggregator restart in the first place.

Two Fluent Bit settings work together here and both matter. storage.max_chunks_up controls how many chunks are memory-mapped and active at once — it governs memory pressure on the agent, not disk usage. storage.total_limit_size is what caps the actual disk consumption of the buffer directory. Set both. Omitting storage.total_limit_size means a prolonged outage can fill your node's disk entirely, which causes a different class of failure.

One more critical piece: monitor your buffer. Alert when fluentbit_output_dropped_records_total increments at all — any non-zero value means messages are being discarded. Also alert when buffer disk usage exceeds 80% of storage.total_limit_size. That's your early warning that the aggregator is falling behind and you need to either scale it or reduce log volume before the hard limit hits.

A practical sizing rule: size your buffer to hold at least 2x the expected throughput during your worst-case outage window. If you normally ship 1 GB/min and your aggregator can be unavailable for up to 10 minutes during a rolling restart, your buffer should comfortably hold 20 GB. Test this explicitly: kill the aggregator in staging, watch the buffer fill, restore the aggregator, and verify zero dropped records in the metrics output.

One detail that catches teams off guard: the buffer path must have sufficient filesystem space and be on a durable volume. If the node itself is ephemeral (like AWS Fargate or GCP Cloud Run), the disk buffer disappears with the node. In those environments, use a network-attached durable buffer like Amazon SQS or Kafka. The principle remains the same — async, durable, monitored.

# Fluent Bit configuration for a Kubernetes DaemonSet log shipping agent.
# This config implements the reliable pipeline pattern:
#   Container stdout → Fluent Bit tail → Disk buffer → Loki (with retry)
#
# Deploy this as a DaemonSet so every node has exactly one agent.
# The agent tails /var/log/containers/* which is where Kubernetes writes
# all container stdout/stderr on the host.

[SERVICE]
    Flush         5
    Grace         30
    HTTP_Server   On
    HTTP_Listen   0.0.0.0
    HTTP_Port     2020
    storage.type              filesystem
    storage.path              /var/log/flb-storage/
    storage.sync              normal
    storage.max_chunks_up     128
    storage.total_limit_size  2G

[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    Parser            docker
    DB                /var/log/flb_kube.db
    Skip_Long_Lines   On
    Refresh_Interval  10

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Merge_Log           On
    Keep_Log            Off
    K8S-Logging.Parser  On

[FILTER]
    Name   grep
    Match  kube.*
    Exclude  log  GET /healthz|GET /readyz|GET /metrics

[OUTPUT]
    Name            loki
    Match           kube.*
    Host            loki.monitoring.svc.cluster.local
    Port            3100
    Labels          job=kubernetes, namespace=$kubernetes['namespace_name'], app=$kubernetes['labels']['app']
    Line_Format     json
    Retry_Limit     5

Retention, Alerting, and the Cost of Keeping Everything Forever

Here's the uncomfortable truth about log storage: keeping every log line forever is not observability — it's hoarding. And it will quietly drain your cloud budget while simultaneously making it harder to find what you're looking for.

A sensible retention strategy is tiered, and the tiers should map to how often you actually query each category of data. Hot storage (Elasticsearch, Loki): last 7 days, indexed and fully queryable, expensive per GB. Warm storage (S3, GCS, queried via Athena or BigQuery): last 90 days, compressed, cheap. Cold/archive (S3 Glacier Instant Retrieval): 1-7 years, for compliance only, query only during audits. The numbers to remember: 80% of your debugging happens within 48 hours of an incident, and most compliance frameworks (PCI DSS, SOC 2, HIPAA) require 1 year of audit log retention. Design your pipeline around those two facts and not around what the default retention setting happened to be when someone first stood up the cluster.

Apply the tiers by log level, not just by age. Debug and trace logs are worthless after 72 hours — they exist to help you understand a problem you're actively investigating. Ship them to S3 after 3 days. Info and warning logs hold their value slightly longer for trend analysis — keep them hot for 7 days, warm for 90. Error logs and explicit audit events (logins, privilege escalations, payment events) have the longest tail — keep them hot for 14 days, warm for 90, cold for up to 7 years depending on your compliance regime.

The second part of this equation is alerting on log content — and here is where teams consistently over-alert. Every ERROR log firing PagerDuty is a recipe for alert fatigue that ends with engineers muting their phones. Alert on derived signals instead: the error rate (errors per minute, not individual errors), the absence of expected business events (zero payment_succeeded events in 10 minutes is far more alarming than a single payment_failed), and sudden cardinality spikes in specific failure reasons. Your aggregator exists to compute these signals — use it.

One more cost-saving pattern worth doing early: pre-aggregate metrics from high-throughput logs. Instead of shipping 50,000 log lines per minute for a busy API endpoint, ship one aggregated record every 10 seconds with request count, error count, and p99 latency. Your alerting pipeline doesn't need every individual request. It needs to know when the shape of traffic changes.

Finally, set an alert on log volume anomalies — specifically, drops. A sudden fall in INFO log volume after a deployment might not mean the system is quiet. It might mean logging is broken. Alert when log volume from any service drops below 20% of its 7-day rolling average for more than 5 minutes. That's the canary that catches a broken logging pipeline before it becomes a silent 20-minute gap.

Also consider cost allocation: tag log streams with a cost centre or team label. Show each team their log storage cost in dollars. That alone reduces volume by 30% in most orgs — teams suddenly realise they don't need debug logs from all 50 microservices retained for 90 days.

# Loki alerting rules using LogQL — Loki's query language.
# These rules run inside Loki's ruler component and fire alerts into
# Alertmanager when conditions are met.
#
# Philosophy: alert on RATES and ABSENCE, not individual error lines.
# One error log is noise. 50 error logs per minute is an incident.
# Zero payment_succeeded events for 5 minutes is a business emergency.

groups:
  - name: payments-service-alerts
    interval: 1m
    rules:
      - alert: PaymentsServiceHighErrorRate
        expr: |
          sum(rate({app="payments-service", namespace="production"} |= `"level":"error"` [1m])) > 10
        for: 2m
        labels:
          severity: critical
          team: payments
        annotations:
          summary: "Payments service error rate is {{ $value | humanize }} errors/min"
          runbook_url: "https://wiki.internal/runbooks/payments-high-error-rate"
          grafana_explore_url: "https://grafana.internal/explore?orgId=1&left=[...]"

      - alert: PaymentsNoSuccessfulTransactions
        expr: |
          sum(rate({app="payments-service", namespace="production"} |= `"payment_succeeded"` [5m])) == 0
        for: 5m
        labels:
          severity: critical
          team: payments
        annotations:
          summary: "No successful payments processed in the last 5 minutes"
          runbook_url: "https://wiki.internal/runbooks/payments-no-transactions"

      - alert: PaymentsGatewayTimeouts
        expr: |
          sum(
            rate(
              {app="payments-service", namespace="production"}
                | json
                | failure_reason = "gateway_timeout"
              [2m]
            )
          ) > 5
        for: 3m
        labels:
          severity: warning
          team: payments
        annotations:
          summary: "Gateway timeout rate: {{ $value | humanize }}/min — possible Stripe outage"
          runbook_url: "https://wiki.internal/runbooks/payments-gateway-timeout"

      - alert: PaymentsLogVolumeAnomaly
        expr: |
          (
            sum(rate({app="payments-service", namespace="production"} [5m]))
            /
            sum(rate({app="payments-service", namespace="production"} [7d]))
          ) < 0.20
        for: 5m
        labels:
          severity: warning
          team: payments
        annotations:
          summary: "Payments log volume is {{ $value | humanizePercentage }} of 7-day average — pipeline may be broken"
          runbook_url: "https://wiki.internal/runbooks/payments-log-volume-drop"

      - alert: FluentBitDroppedRecords
        expr: |
          increase(fluentbit_output_dropped_records_total[2m]) > 0
        for: 1m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: "Fluent Bit is dropping records — log data is being permanently lost"
          runbook_url: "https://wiki.internal/runbooks/fluentbit-dropped-records"

      - alert: FluentBitBufferDiskHighUsage
        expr: |
          (fluentbit_storage_chunks_size_bytes / fluentbit_storage_chunks_size_bytes_limit) > 0.80
        for: 5m
        labels:
          severity: warning
          team: platform
        annotations:
          summary: "Fluent Bit buffer at {{ $value | humanizePercentage }} of disk limit — aggregator may be falling behind"
          runbook_url: "https://wiki.internal/runbooks/fluentbit-buffer-high"

Choosing Your Log Aggregation Stack: ELK vs Loki vs CloudWatch

You can't choose a log aggregation tool purely on features — every choice is a trade-off between cost, query speed, and operational complexity. The three most common production stacks in 2026 are ELK (Elasticsearch + Logstash + Kibana), Grafana Loki, and cloud-native solutions like AWS CloudWatch Logs. Each has a natural home. Picking the wrong one for your context is an expensive mistake to undo.

ELK is the most feature-rich. It full-text indexes every field at ingest time, so any substring search across any field is fast. That power has a price: the index itself is large, SSD-backed, and expensive. ELK at 10 TB/day costs tens of thousands of dollars monthly in cluster nodes, and it needs a dedicated ops team to tune shard counts, manage JVM heap, and handle cluster splits during rolling upgrades. ELK shines in compliance-heavy environments (PCI, HIPAA, FedRAMP) where you need fast, full-text audit trail queries and where the cost is justified by regulatory necessity.

Loki flips the model. It only indexes the labels you define (like Prometheus does for metrics), and stores log content as compressed chunks in object storage — S3, GCS, or Azure Blob. This makes Loki 5 to 10 times cheaper at equivalent volumes compared to ELK. The trade-off is query performance on unindexed fields: if you query over a large time range without narrowing by a label first, Loki has to scan compressed chunks, which is slower. The discipline is to design your queries around labels for the initial filter, then use | json to filter on structured fields within those results. Loki is the natural fit for cloud-native microservices in Kubernetes, especially if Grafana is already your dashboarding layer.

CloudWatch Logs is the simplest entry point: no agents to deploy if you're on Lambda or ECS with the AWS log driver, pay-per-ingest pricing, and native integration with CloudWatch Metrics and Alarms. The ceiling appears quickly though. Cross-account log queries are painful. Exporting data out of AWS costs $0.09/GB in egress. CloudWatch Insights queries over large time ranges can be slow and expensive. CloudWatch is the right starting point for small-to-medium AWS-native workloads where the team has no dedicated SRE and simplicity is worth the per-GB premium.

Your decision comes down to four factors: daily volume, query patterns, operational capacity, and budget. The right stack is the one your team can operate at full fidelity, with no corners cut on retention, without burning engineering time keeping it alive.

A rule of thumb from several migrations: under 200 GB/day on AWS with no dedicated SRE, start with CloudWatch. In Kubernetes with Grafana already deployed, start with Loki. If you have compliance requirements that mandate full-text audit trails or if daily volume exceeds 2 TB, evaluate ELK — but get an Elasticsearch specialist involved before you commit.

On the managed vs self-hosted question: managed versions (Elastic Cloud, Grafana Cloud, CloudWatch) eliminate operational toil but carry a per-GB premium of 2 to 4 times the self-hosted compute cost. For most teams, managed is the correct call until daily volume consistently exceeds 5 TB. Below that threshold, the engineering hours saved by not running Elasticsearch or Loki yourself are worth more than the cost delta.

One more aspect: lock-in. CloudWatch and Grafana Cloud tie you to their ecosystem. Migrating away is expensive. ELK is open-source (with Elastic's licensing nuance). Loki is fully open-source under AGPL. If you value flexibility, prefer open-source stacks from day one.

package io.thecodeforge.logging;

import java.util.*;

public class LogDecisionEngine {

    public enum LogStack { ELK, LOKI, CLOUDWATCH }

    public static class Requirements {
        final long dailyVolumeGB;
        final boolean requiresFullTextSearch;
        final boolean kubernetesNative;
        final boolean awsLocked;
        final int opsHeadcount;

        public Requirements(long dailyVolumeGB, boolean requiresFullTextSearch,
                            boolean kubernetesNative, boolean awsLocked, int opsHeadcount) {
            this.dailyVolumeGB = dailyVolumeGB;
            this.requiresFullTextSearch = requiresFullTextSearch;
            this.kubernetesNative = kubernetesNative;
            this.awsLocked = awsLocked;
            this.opsHeadcount = opsHeadcount;
        }
    }

    public static LogStack decide(Requirements req) {
        if (req.dailyVolumeGB < 100 && req.awsLocked && req.opsHeadcount < 2) {
            return LogStack.CLOUDWATCH;
        }
        if (req.requiresFullTextSearch || req.dailyVolumeGB > 500) {
            return LogStack.ELK;
        }
        if (req.kubernetesNative && req.dailyVolumeGB > 50) {
            return LogStack.LOKI;
        }
        return req.kubernetesNative ? LogStack.LOKI : LogStack.ELK;
    }

    public static void main(String[] args) {
        Requirements typicalK8s = new Requirements(300, false, true, false, 2);
        System.out.println("Typical K8s platform:  " + decide(typicalK8s));
        Requirements complianceEcom = new Requirements(800, true, true, false, 4);
        System.out.println("Compliance e-commerce: " + decide(complianceEcom));
        Requirements smallAws = new Requirements(20, false, false, true, 1);
        System.out.println("Small AWS startup:     " + decide(smallAws));
    }
}

Compliance and Audit Logging: What PCI DSS Actually Requires

The title incident — the 20-minute gap that cost a PCI audit — happened because the team didn't understand what PCI DSS requirement 10 actually demands. It's not just 'keep logs'. It's: 'implement audit trails that link all access to individual users, retain them for at least one year, and monitor for anomalies.' The gap meant three months of re-audit work and a fine. Here's what you need to know.

PCI DSS Requirement 10 specifically requires: 10.2 (audit trails for all access to cardholder data), 10.3 (record at least user ID, event type, date/time, success/failure, origination, identity of affected data), 10.5 (protect audit trails from modification), 10.6 (review logs daily), 10.7 (retain audit trail history for at least one year, with three months immediately available online). The critical detail: logs must be immutable after generation. A misconfigured buffer that drops logs violates 10.5 — your auditor will fail you.

To meet these requirements, your logging pipeline must guarantee: no gaps (disk-backed buffer), no tampering (write-once storage with access controls), no manual review overload (automated alerting on anomalies), and retention that spans the full year with the last 3 months hot-queryable. Most teams fail on the 'immediately available online' part — they archive everything to cold storage after 7 days, but PCI wants 3 months of hot data for daily reviews.

Design your retention tiers accordingly: hot (Loki or Elasticsearch) for latest 90 days, warm (S3/Athena) for months 4-12, cold (Glacier) for years 2-7 if you keep beyond PCI. The hot tier must support daily log review queries — a single day's logs for all payment-related services should return in under 30 seconds. If it takes minutes, your daily review process collapses.

One more thing: access control on logs. PCI 10.5 requires that logs cannot be modified or deleted. Your storage backend must enforce immutability. In Loki, use the single-store mode with object storage that has versioning enabled. In Elasticsearch, disable index deletion for audit indexes and use index lifecycle management with a lock. In CloudWatch, log group policies prevent deletion by non-admin roles but can still be truncated by retention settings — set retention to never expire for audit log groups and export to S3 with object lock.

Finally, the daily review (10.6) must be automated. No one reads 10 GB of logs per day manually. Use the alerting patterns from the previous section — error rate anomalies, absence of expected events, and log volume drops. Your auditor will ask for proof that these alerts exist and have runbooks. Build them before the audit, not after.

# Loki retention policy for PCI compliance.
# Ensures 90 days hot retention and 1 year total with immutability.

# Define custom retention for the 'audit' bucket
# All payment-related logs should be sent to this bucket via Loki distributor.

tenant: "my-org"

schema_config:
  configs:
    - from: 2026-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: audit_
        period: 24h

storage_config:
  tsdb_shipper:
    active_index_directory: /loki/tsdb-shipper/active
    cache_location: /loki/tsdb-shipper/cache
    cache_ttl: 24h
    shared_store: s3
  aws:
    bucket: my-org-audit-logs
    endpoint: s3.eu-west-1.amazonaws.com
    region: eu-west-1
    access_key_id: ${AWS_ACCESS_KEY_ID}
    secret_access_key: ${AWS_SECRET_ACCESS_KEY}

limits_config:
  retention_period: 365d  # 1 year total retention
  retention_stream:
    - selector: '{namespace="payment", app=~".+-service"}'
      priority: 1
      period: 90d  # Hot retention for these streams

compactor:
  working_directory: /loki/compactor
  shared_store: s3
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150

# Enforce immutability: Loki does not allow deletion by default.
# Ensure S3 bucket has object lock enabled.
# In AWS, add bucket policy denying s3:DeleteObject for the Loki user.

Guard The Perimeter: Why Centralisation Without Isolation Fails

Every log pipeline assumes its sources are trustworthy. That assumption costs you a pager at 3am.

A misconfigured container spamming ERROR messages at 10,000 writes per second will saturate your ingestion API. Your cheap logging agent on a memory-constrained microVM crashes when its buffer fills. Then you lose production logs for every other service behind the same collector.

The fix is queue isolation per namespace or criticality tier. Production payment services should never share a log forwarder buffer with a staging cron job that runs database migrations and prints debug output. Use separate kafka topics, distinct CloudWatch log groups with per-stream throttling, or dedicated fluentd instances with independent backpressure config.

Enforce rate limits at the network edge for each source. A spike in a single service's log volume should degrade only that service's observability, not the entire fleet's. Isolation buys you blast radius control. Without it, your aggregation stack is one runaway loop from going blind.

// io.thecodeforge — devops tutorial

fluentd:
  workers: 4
  buffer:
    type: file
    path: /var/log/fluent/buffers
    total_limit_size: 1GB
  sources:
    - @type: tail
      path: /var/log/containers/production/*.log
      tag: "production.${tag}"
      read_lines_limit: 1000
    - @type: tail
      path: /var/log/containers/staging/*.log
      tag: "staging.${tag}"
      read_lines_limit: 100
  matches:
    - pattern: "production.**"
      @type: kafka
      brokers: kafka-cluster:9092
      topic: logs-production
      buffer:
        chunk_limit_size: 4MB
        flush_interval: 1s
    - pattern: "staging.**"
      @type: kafka
      brokers: kafka-cluster:9092
      topic: logs-staging
      buffer:
        chunk_limit_size: 512KB
        flush_interval: 10s

Replay Is Your Safety Net When The Pipeline Burns Down

Your aggregation pipeline will fail. A disk fills, a network partition splits your collectors, or your sink goes read-only after a cloud provider incident. The question is not if, but how fast you restore continuity.

Replay readiness is the difference between a five minute gap and a five hour fire drill. Every log agent should buffer to local disk with a survival time that exceeds your maximum outage window. We run file-based buffers with a 48-hour retention for production logs. That buys us time to fix the pipeline, then reprocess the dead letter queue via a simple tail of the buffer files.

The pattern is idempotent: you replay the same bytes, the sink deduplicates on the log event ID you embedded at creation time. Test your replay path monthly. Send a batch of test events, kill your collector, restart it, and verify no events were lost and none duplicated. If that test takes more than an hour to run, your buffer config is too brittle.

Do not treat log shipping as fire-and-forget. Treat it as an at-least-once delivery system with local persistence. Your future self, debugging a midnight incident, will thank you.

// io.thecodeforge — devops tutorial

fluentd:
  buffer:
    type: file
    path: /var/log/fluent/buffer
    retry_forever: false
    retry_max_times: 3
    retry_wait: 10s
    retry_exponential_backoff: true
    overflow_action: block
  dead_letter:
    path: /var/log/fluent/dead_letters
    max_files: 10
    max_file_size: 100MB
  replay:
    script: |
      for file in /var/log/fluent/buffer/*.buffer; do
        tail -n +0 "$file" | fluent-cat log.replay --host localhost --port 24224
      done