Cloud Logging: Log Router, Log Buckets, and Log-Based Metrics

You're paged at 3 AM because your API latency spiked. You SSH into a box, grep through gigabytes of logs, and find nothing. Meanwhile, your competitor's SRE already identified the root cause via a log-based metric that triggered an alert. That's the difference Cloud Logging makes. It's not just about storing logs—it's about turning them into actionable signals. In this post, we'll dissect the three pillars: Log Router to control log flow, Log Buckets for cost-effective storage, and Log-Based Metrics to create custom monitoring without code changes. By the end, you'll know how to build a logging pipeline that doesn't just collect data but actively helps you sleep better at night.

Log Router: The Traffic Cop for Your Logs

The Log Router is the entry point for all log entries in Cloud Logging. Every log—whether from Compute Engine, Kubernetes, or your app—passes through the router before being stored or excluded. Its primary job is to match logs against inclusion and exclusion filters, then route matched logs to one or more sinks. Sinks can be Log Buckets (default), Pub/Sub topics, BigQuery datasets, or Cloud Storage buckets. This is where you control costs: by excluding verbose logs (e.g., health checks) or routing audit logs to a cheap storage class. The router uses a first-match wins logic: if a log matches an exclusion filter, it's dropped; otherwise, it's evaluated against inclusion filters. Misconfiguring these filters is a common cause of missing logs during incidents. Always test filters with the Logs Explorer before deploying.

gcloud logging sinks create my-bigquery-sink \
    bigquery.googleapis.com/projects/my-project/datasets/audit_logs \
    --log-filter='resource.type="k8s_cluster" AND severity>=WARNING' \
    --description="Route K8s warnings+ to BigQuery"

Log Buckets: Storage with Retention and Access Control

Log Buckets are the default storage destination for logs routed by the Log Router. Each bucket has a retention period (default 30 days, max 3650 days) and can be locked to prevent deletion. Buckets are regional, so choose a region close to your compute resources to reduce egress costs. You can create custom buckets with different retention policies—for example, a 7-day bucket for debug logs and a 365-day bucket for audit logs. Buckets also support CMEK (Customer-Managed Encryption Keys) for compliance. Access to bucket contents is controlled via IAM roles like Logs Viewer and Logs Bucket Writer. Important: deleting a bucket deletes all logs inside, even if retention hasn't expired. Always use retention policies and bucket locks for critical logs.

gcloud logging buckets create audit-bucket \
    --location=us-central1 \
    --retention-days=365 \
    --description="Long-term audit log storage" \
    --locked

Log-Based Metrics: Turn Logs into Monitoring Signals

Log-Based Metrics allow you to count log entries matching a filter and expose them as Cloud Monitoring metrics. This is incredibly powerful because you can create custom metrics without modifying application code. For example, count every ERROR log line from your payment service and alert when the rate exceeds 5 per minute. There are two types: counter metrics (count of log entries) and distribution metrics (histogram of a numeric value extracted from logs). Distribution metrics require a label extractor using regular expressions. Metrics are created via the Logs Explorer or API, and they appear in Metrics Explorer within minutes. They incur no additional cost beyond the underlying log storage. However, beware of high-cardinality labels (e.g., request IDs) which can spike costs and degrade performance.

gcloud logging metrics create payment-error-rate \
    --description="Count of ERROR logs in payment service" \
    --log-filter='resource.type="k8s_container" AND resource.labels.container_name="payment" AND severity=ERROR'

Routing Logs to Multiple Destinations

A single log entry can be routed to multiple sinks. This is useful when you need logs in both a Log Bucket for real-time analysis and BigQuery for long-term analytics. To achieve this, create multiple sinks with overlapping filters. However, be mindful of costs: each sink that matches a log entry incurs ingestion charges (though egress to BigQuery or Storage is free within the same region). You can also route logs to Pub/Sub for streaming to third-party tools like Splunk or Datadog. When routing to Pub/Sub, ensure the topic has sufficient quota and that subscribers are resilient to backpressure. A common pattern is to route all logs to a default bucket, then route a subset (e.g., errors) to a Pub/Sub topic for real-time alerting.

gcloud logging sinks create error-pubsub \
    pubsub.googleapis.com/projects/my-project/topics/error-logs \
    --log-filter='severity>=ERROR' \
    --description="Route errors to Pub/Sub"

gcloud logging sinks create all-logs-bucket \
    logging.googleapis.com/projects/my-project/locations/global/buckets/_Default \
    --log-filter='' \
    --description="Default sink for all logs"

Exclusion Filters: Cutting Costs Without Losing Visibility

Exclusion filters are applied before any sink evaluation. If a log matches an exclusion filter, it is dropped entirely and never stored. This is the most effective way to reduce logging costs, but it's also dangerous. Common exclusions include health check logs, repetitive debug logs, and known benign errors. However, if you exclude too aggressively, you might miss anomalies. Best practice: exclude only after analyzing log patterns over a week, and always log excluded entries to a separate bucket for a trial period. You can also use exclusion filters with sampling: for example, exclude 90% of INFO logs but keep all ERROR logs. Exclusion filters are evaluated in order, and the first match wins.

gcloud logging sinks update _Default \
    --add-exclusion-filter='resource.type="k8s_container" AND resource.labels.container_name="health-check"' \
    --add-exclusion-name='exclude-health-checks'

IAM and Access Control for Logs

Access to Cloud Logging resources is governed by IAM roles. Key roles include: Logs Viewer (read logs), Logs Writer (write logs), Logs Bucket Writer (write to a specific bucket), and Logs Configuration Writer (manage sinks and metrics). For production, follow least privilege: give developers Logs Viewer on specific buckets, not the entire project. Service accounts that write logs (e.g., from GKE) need Logs Writer. Audit logs (Cloud Audit Logs) are stored in the _Required bucket and are accessible only to users with the Project Owner or Logs Viewer role at the organization level. Be careful with custom roles: missing permissions can cause silent log drops.

gcloud projects add-iam-policy-binding my-project \
    --member='user:dev@example.com' \
    --role='roles/logging.viewer' \
    --condition='resource.name.startsWith("projects/my-project/locations/global/buckets/audit-bucket")'

Logs Explorer: Querying Logs Efficiently

The Logs Explorer is the UI for searching and analyzing logs. It supports a SQL-like query language with operators like =, !=, AND, OR, and functions like timestamp() and severity(). You can filter by resource type, log name, or any structured field. For performance, always include a time range and avoid wildcard searches on large fields. Use the 'Show query' button to see the underlying filter syntax, which you can reuse in sinks and metrics. The Logs Explorer also supports histogram views to visualize log volume over time. For advanced analysis, you can export query results to BigQuery. Pro tip: save common queries as 'Saved Searches' for team reuse.

resource.type="k8s_container"
resource.labels.container_name="payment"
severity>=ERROR
timestamp>="2026-07-11T00:00:00Z"
|> count by severity

Monitoring and Alerting with Log-Based Metrics

Once a log-based metric is created, it appears in Cloud Monitoring as a custom metric. You can create alerting policies based on these metrics, such as 'alert if payment-error-rate > 10 per minute for 5 minutes'. The metric is available in Metrics Explorer and can be used in dashboards. Log-based metrics are real-time (latency < 1 minute) and incur no additional cost beyond the underlying log storage. However, be aware that metric creation is eventually consistent—it may take a few minutes for new logs to appear in the metric. Also, if you delete the log-based metric, the underlying logs are not deleted; only the metric stops updating.

gcloud alpha monitoring policies create \
    --condition-filter='metric.type="logging.googleapis.com/user/payment-error-rate" AND metric.labels.severity="ERROR"' \
    --condition-threshold-value=10 \
    --condition-threshold-duration=300s \
    --notification-channels='projects/my-project/notificationChannels/123' \
    --display-name='Payment Error Rate Alert'

Cost Management: Understanding Logging Pricing

Cloud Logging pricing is based on three components: ingestion (volume of log data), storage (retained data), and egress (data exported to other destinations). Ingestion is charged per GiB, with a free tier of 50 GiB per project per month. Storage costs depend on retention period and bucket location. Exclusions reduce ingestion costs, while shorter retention reduces storage costs. Log-based metrics and Logs Explorer queries are free. To optimize costs: exclude verbose logs, use shorter retention for debug buckets, and route audit logs to cheaper storage classes (e.g., Cloud Storage Nearline). Monitor your logging costs in the Billing console and set budget alerts.

# Estimate monthly logging cost for 100 GiB ingestion, 30-day retention
# Ingestion: 100 GiB * $0.50/GiB = $50
# Storage: 100 GiB * 30 days * $0.01/GiB/day = $30
# Total: $80/month

Troubleshooting Common Logging Issues

Common issues include: logs not appearing in Logs Explorer, sinks not delivering logs, and log-based metrics not showing data. First, check if the log entry exists by using the Logs Explorer with a broad filter. If logs are missing, verify that the source (e.g., GKE, VM) has the correct IAM permissions to write logs. For sinks, check the sink's 'writer identity' and ensure the destination (e.g., BigQuery dataset) exists and has proper permissions. Use the 'Logs Router' page to see sink errors. For log-based metrics, ensure the metric filter matches the log entries and that the metric has been created in the correct project. Also, remember that metrics are eventually consistent—wait a few minutes.

gcloud logging sinks describe my-sink \
    --format='json' | jq '.writerIdentity'

Best Practices for Production Logging Pipelines

1. Use structured logging (JSON) to enable easy filtering and metric extraction. 2. Set log levels consistently across services. 3. Route audit logs to a locked bucket with long retention. 4. Exclude health check and verbose debug logs at the router level. 5. Create log-based metrics for all critical error conditions. 6. Monitor sink health and set alerts for sink failures. 7. Use IAM conditions to restrict access to sensitive logs. 8. Regularly review logging costs and adjust exclusions. 9. Test your logging pipeline during load testing to ensure it doesn't become a bottleneck. 10. Document your logging architecture, including sink configurations and metric definitions.

logging:
  sinks:
    - name: default-bucket
      destination: logging.googleapis.com/projects/my-project/locations/global/buckets/_Default
      filter: ""
    - name: error-pubsub
      destination: pubsub.googleapis.com/projects/my-project/topics/error-logs
      filter: "severity>=ERROR"
  exclusions:
    - name: exclude-health-checks
      filter: "resource.type=\"k8s_container\" AND resource.labels.container_name=\"health-check\""
  metrics:
    - name: payment-error-rate
      filter: "resource.type=\"k8s_container\" AND resource.labels.container_name=\"payment\" AND severity=ERROR"

Observability Analytics: SQL Queries on Your Logs

The Logs Explorer is great for real-time debugging, but for complex analysis across large time ranges, you need SQL. Cloud Logging's Observability Analytics (powered by BigQuery) lets you upgrade a log bucket to enable SQL queries directly on your log data. Once enabled, you can run standard SQL queries against your logs using the Log Analytics page — no need to export to BigQuery first. This enables joins across log types, aggregation windows, percentile calculations, and other SQL patterns that the Logs Explorer's filter language can't express. For example: 'What is the p99 latency of my payment service per deployment over the last 30 days?' or 'Which service has the highest error rate per region?' You can also create linked BigQuery datasets for deeper analysis or integration with existing BI tools. Upgrading a bucket to use Observability Analytics takes a few minutes and does not affect existing log ingestion. Only new logs after the upgrade are queryable via SQL. Use Observability Analytics for postmortem analysis, trend detection, and compliance reporting. In production, we run a weekly SQL query that compares error rates across all services — it caught a gradual degradation that the Logs Explorer dashboard missed.

# Enable analytics on a bucket first:
# gcloud logging buckets update my-bucket --location=us-central1 --enable-analytics

# Then query logs with SQL:
SELECT
  resource.labels.container_name AS service,
  COUNT(*) AS error_count,
  APPROX_QUANTILES(CAST(json_payload.latency AS INT64), 100)[OFFSET(99)] AS p99_latency_ms
FROM
  `my-project.us-central1.my-bucket._allLogs`
WHERE
  severity = 'ERROR'
  AND timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
GROUP BY
  service
ORDER BY
  error_count DESC

Log Views and Linked Datasets: Granular Access Control and Analytics

By default, anyone with Logs Viewer on a project can see all logs in all buckets. Log Views let you define subsets of logs within a bucket and grant access to specific views instead of the entire bucket. For example, create a view for 'payment-service-errors' that only includes ERROR logs from the payment service, then grant the payments team access to only that view. This enables multi-tenant log access without creating separate buckets. Additionally, you can create a linked dataset in BigQuery from an upgraded log bucket (one with Observability Analytics enabled). This creates a BigQuery dataset that mirrors your log bucket, allowing you to run BigQuery's full analytical capabilities (including federated queries, scheduled queries, and BI tool integration) without double storage costs. The linked dataset is read-only and stays in sync with the log bucket. In production, we use log views to give developers access to their service's logs without exposing other teams' data, and linked datasets for long-term compliance analytics in BigQuery.

# Create a log view that only shows payment service errors
gcloud logging views create payment-errors \
  --bucket=my-bucket \
  --location=us-central1 \
  --log-filter='resource.labels.container_name="payment" AND severity>=ERROR'

# Grant access to the payments team
gcloud logging views add-iam-policy-binding payment-errors \
  --bucket=my-bucket \
  --location=us-central1 \
  --member='group:payments-team@example.com' \
  --role='roles/logging.viewer'

# Create a linked BigQuery dataset for analytics
gcloud logging linked-datasets create my-bq-link \
  --bucket=my-bucket \
  --location=us-central1

echo "Linked dataset created. Query at: my-project.us-central1.my-bq-link"