Google Cloud — Cloud Trace (Distributed Tracing)

Cloud Trace is a distributed tracing service that collects latency data from your applications and surfaces performance bottlenecks. It integrates with OpenTelemetry and Google Cloud's client libraries to automatically trace gRPC and HTTP requests. Effective sampling and analysis helps identify the root cause of slow requests without overwhelming storage costs.

Why Distributed Tracing Matters in Production

In a monolithic application, a single request's lifecycle is easy to follow. But in microservices, a single user action can fan out to dozens of services, queues, and databases. When latency spikes or errors occur, traditional logging gives you isolated snapshots — you can't connect the dots. Distributed tracing solves this by propagating a unique trace ID across every hop, allowing you to reconstruct the full request path. Google Cloud Trace is a managed distributed tracing service that integrates natively with Google Cloud services like Cloud Run, GKE, and App Engine, and can also ingest traces from any OpenTelemetry-compatible source. Without tracing, you're flying blind in a distributed system. With it, you can pinpoint the exact service causing a slowdown, identify cascading failures, and optimize critical paths. In production, we've seen teams spend weeks debugging a 500ms latency regression that tracing revealed in minutes: a single Redis call that was accidentally hitting a cross-region replica.

# Enable Cloud Trace API
gcloud services enable cloudtrace.googleapis.com

# For GKE, ensure workload has permission
# Create a service account with roles/cloudtrace.agent

gcloud iam service-accounts create trace-sa \
    --display-name="Cloud Trace Service Account"

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
    --member="serviceAccount:trace-sa@${PROJECT_ID}.iam.gserviceaccount.com" \
    --role="roles/cloudtrace.agent"

# Annotate GKE node pool to use this SA
kubectl annotate serviceaccount default \
    iam.gke.io/gcp-service-account=trace-sa@${PROJECT_ID}.iam.gserviceaccount.com

OpenTelemetry: The Universal Instrumentation Standard

Google Cloud Trace supports multiple ingestion formats, but OpenTelemetry (OTel) is the future-proof choice. OTel provides vendor-neutral APIs and SDKs for generating traces, metrics, and logs. By instrumenting your services with OTel, you avoid lock-in and can switch backends (e.g., from Cloud Trace to Jaeger) without code changes. The OTel Collector acts as a gateway: it receives traces from your services, processes them (sampling, batching, adding attributes), and exports to Cloud Trace via the Google Cloud exporter. In production, we always run the OTel Collector as a sidecar or DaemonSet to offload export logic from application code. This also allows centralized sampling decisions — you can trace 100% of requests for critical services and 1% for high-volume ones. The key is to propagate context correctly: use W3C TraceContext headers (traceparent, tracestate) across all service boundaries. Without proper propagation, traces break into disconnected fragments.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  googlecloud:
    project: my-production-project
    # Use service account credentials
    # Default: ADC (Application Default Credentials)
  logging:
    loglevel: debug

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
  probabilistic_sampler:
    hash_seed: 42
    sampling_percentage: 10

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, probabilistic_sampler]
      exporters: [googlecloud, logging]

Instrumenting a Go Service with OpenTelemetry

Let's instrument a production Go HTTP service. We'll use the official OpenTelemetry Go SDK and the Google Cloud exporter. The key steps: create a tracer provider, configure the exporter, and wrap HTTP handlers to automatically create spans. We'll also propagate context via HTTP headers. In production, you should never manually create spans for every function — use automatic instrumentation where possible. For Go, the otelhttp middleware handles incoming requests, and you can create child spans for internal operations like database calls or RPCs. Always set attributes like http.method, http.url, and http.status_code — these are used by Cloud Trace's built-in dashboards. Also, set service.name as a resource attribute to distinguish between services. Without proper resource attributes, all traces appear under a single unnamed service in Cloud Trace, making filtering impossible.

package main

import (
	"context"
	"log"
	"net/http"
	"os"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
)

func initTracer() func() {
	ctx := context.Background()
	exporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithEndpoint("otel-collector:4317"), otlptracegrpc.WithInsecure())
	if err != nil {
		log.Fatalf("failed to create exporter: %v", err)
	}
	res, err := resource.New(ctx,
		resource.WithAttributes(
			semconv.ServiceNameKey.String("payment-service"),
			attribute.String("environment", "production"),
		),
	)
	if err != nil {
		log.Fatalf("failed to create resource: %v", err)
	}
	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(res),
		sdktrace.WithSampler(sdktrace.AlwaysSample()),
	)
	otel.SetTracerProvider(tp)
	return func() { tp.Shutdown(ctx) }
}

func main() {
	cleanup := initTracer()
	defer cleanup()

	handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		// Simulate work
		w.Write([]byte("Hello, traced world!"))
	})
	wrapped := otelhttp.NewHandler(handler, "hello")
	http.Handle("/hello", wrapped)
	log.Fatal(http.ListenAndServe(":8080", nil))
}

Sampling Strategies: Balancing Cost and Visibility

Tracing every request in a high-throughput system is expensive and generates massive data volumes. Cloud Trace charges based on ingested spans, and excessive tracing can lead to runaway costs. The solution is sampling. There are two main strategies: head-based sampling (decide at the start of a trace) and tail-based sampling (decide after seeing the full trace). Head-based is simpler and supported by OTel's probabilistic sampler. Tail-based is more accurate but requires a collector that buffers traces. In production, we use a hybrid: head-based sampling for most services (e.g., 1-10%), but for critical services like checkout or authentication, we use identity-based sampling to trace 100% of requests from specific users or error codes. Cloud Trace also supports adaptive sampling via the googlecloud exporter, which automatically adjusts rates based on budget. However, we've found it unpredictable — we prefer explicit configuration. Always monitor your span ingestion rate and set budget alerts.

processors:
  probabilistic_sampler:
    hash_seed: 42
    sampling_percentage: 5
  # Tail-based sampling (requires experimental processor)
  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    expected_new_traces_per_sec: 10
    policies:
      - name: error-policy
        type: status_code
        status_code: ERROR
        sampling_percentage: 100
      - name: slow-policy
        type: latency
        threshold_ms: 500
        sampling_percentage: 100
      - name: default-policy
        type: probabilistic
        sampling_percentage: 1

Analyzing Traces in Cloud Trace: The UI and API

Once traces are ingested, Cloud Trace provides a web UI and an API for analysis. The Trace List view shows all traces matching a filter — you can filter by service, latency, date, and annotations. The Trace Details view shows a waterfall of spans, with each span's duration, status, and attributes. Key features: you can compare traces side-by-side, create custom dashboards, and set up alerts based on latency thresholds. The API allows programmatic access for custom analysis — for example, you can query traces with errors and correlate them with logs. In production, we use the Trace List to identify the slowest endpoints daily. We also set up a Cloud Monitoring alert that triggers when p99 latency exceeds 500ms for any service. The alert includes a link to the Trace List pre-filtered for that service and time range. This reduces mean time to detection (MTTD) from hours to minutes.

# List traces in the last hour with latency > 1s
gcloud trace traces list \
    --project=my-production-project \
    --start-time="$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)" \
    --end-time="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
    --filter="span.duration > 1s" \
    --format="json"

# Get trace details by trace ID
gcloud trace traces describe TRACE_ID \
    --project=my-production-project \
    --format="json"

Correlating Traces with Logs and Metrics

Traces alone are powerful, but their true value emerges when combined with logs and metrics. Google Cloud's operations suite (formerly Stackdriver) allows you to correlate traces with logs via the trace field in log entries. When you emit a log entry with the trace ID, you can jump from a trace span to the relevant logs. Similarly, you can create metrics from trace data, such as request latency per service. In production, we enforce that every service must include the trace ID in all log entries. We use structured logging (JSON) with a logging.googleapis.com/trace field. This allows us to click from a trace span to the exact log lines that occurred during that span. Without this correlation, you waste time switching between tools. We also export trace metrics to Cloud Monitoring to create dashboards showing p50, p95, and p99 latency per service over time.

import (
	"context"
	"log"
	"os"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/trace"
	"cloud.google.com/go/logging"
)

var logger *logging.Logger

func initLogging(ctx context.Context) {
	client, err := logging.NewClient(ctx, os.Getenv("PROJECT_ID"))
	if err != nil {
		log.Fatalf("Failed to create logging client: %v", err)
	}
	logger = client.Logger("my-service-log")
}

func logWithTrace(ctx context.Context, msg string) {
	span := trace.SpanFromContext(ctx)
	traceID := span.SpanContext().TraceID().String()
	logger.Log(logging.Entry{
		Payload:  map[string]interface{}{"message": msg, "trace": traceID},
		Severity: logging.Info,
		Trace:    traceID,
	})
}

Advanced: Custom Spans and Annotations

Automatic instrumentation covers HTTP and gRPC, but you often need to trace internal operations like database queries, cache calls, or async work. OpenTelemetry allows you to create custom spans manually. You can also add annotations (key-value pairs) and events (timestamps with messages) to spans. In production, we use custom spans to trace every external dependency call: Redis, Pub/Sub, Cloud SQL, etc. We also add annotations for business context like user.id, order.id, and error.details. This makes traces searchable and actionable. For example, we can filter all traces where order.id equals a specific value to debug a user's issue. However, be careful not to add too many attributes — Cloud Trace has a limit of 128 attributes per span. We prioritize attributes that are useful for filtering and debugging.

import (
	"context"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/codes"
)

func processOrder(ctx context.Context, orderID string) error {
	tracer := otel.Tracer("order-service")
	ctx, span := tracer.Start(ctx, "processOrder",
		attribute.String("order.id", orderID),
	)
	defer span.End()

	// Simulate work
	time.Sleep(100 * time.Millisecond)

	// Add event
	span.AddEvent("order-validated",
		attribute.String("validation", "passed"),
	)

	// Simulate error
	if orderID == "bad" {
		span.SetStatus(codes.Error, "invalid order")
		span.RecordError(fmt.Errorf("invalid order ID"))
		return fmt.Errorf("invalid order")
	}

	return nil
}

Troubleshooting Common Trace Issues

Even with proper instrumentation, traces can fail silently. Common issues: missing trace context propagation (e.g., not forwarding headers), misconfigured exporters, and sampling dropping important traces. In production, we have a checklist for debugging missing traces: 1) Verify the OTel Collector is running and receiving spans (check its logs). 2) Check that the service account has cloudtrace.agent role. 3) Ensure the traceparent header is present in outgoing requests. 4) Verify the collector's exporter is configured with the correct project ID. 5) Check Cloud Trace's 'Ingestion' tab for errors. We also run a synthetic health check that sends a traced request every minute and alerts if the trace doesn't appear in Cloud Trace within 30 seconds. This catches configuration drift early. Another common issue is span ID collision — extremely rare but possible. OpenTelemetry uses 64-bit span IDs, so collisions are unlikely but can cause trace merging.

# Check collector logs for errors
kubectl logs -l app=otel-collector --tail=50 | grep -i error

# Verify trace header propagation
curl -v http://my-service/hello 2>&1 | grep -i traceparent

# Check Cloud Trace ingestion metrics
gcloud monitoring metrics list \
    --filter="metric.type = cloudtrace.googleapis.com/span/ingested_count" \
    --format="json"

Cost Management and Budget Alerts

Cloud Trace pricing is based on the number of ingested spans. The first 250,000 spans per month are free, then $0.10 per 100,000 spans. For high-throughput services, costs can escalate quickly. We recommend: 1) Set a monthly budget alert at 80% of your expected spend. 2) Use sampling aggressively — 1% is often enough for high-volume services. 3) Exclude health check endpoints from tracing (they generate noise). 4) Use the OTel Collector's filter processor to drop spans from non-critical services. 5) Monitor your span ingestion rate via Cloud Monitoring. In production, we have a dashboard that shows daily span count per service. If a service's count spikes, we investigate. We also enforce a maximum sampling rate via a centralized OTel configuration that developers cannot override. This prevents accidental cost explosions.

processors:
  filter:
    spans:
      include:
        match_type: regexp
        services:
          - ".*/healthz"
      exclude:
        match_type: strict
        services:
          - "health-check-service"
  probabilistic_sampler:
    sampling_percentage: 1

exporters:
  googlecloud:
    project: my-production-project
    # Set a budget hint (not enforced by Google)
    # Use client-side rate limiting
    retry_on_failure:
      enabled: true
      max_elapsed_time: 60s

Integrating Cloud Trace with CI/CD Pipelines

Tracing isn't just for runtime — you can use it in CI/CD to catch performance regressions before they reach production. By running a load test in a staging environment with tracing enabled, you can compare latency distributions between builds. Cloud Trace's API allows you to query traces by tags — we tag each deployment with the commit SHA and environment. Then, in CI, we run a script that queries traces from the last 10 minutes and compares p99 latency to the baseline. If latency increases by more than 10%, the pipeline fails. This catches regressions early. In production, we also use trace data to validate canary deployments: we compare trace metrics between the old and new versions. If the new version shows higher latency or error rates, the canary is rolled back automatically.

#!/bin/bash
# Run after deploying to staging
COMMIT_SHA=$(git rev-parse HEAD)

# Wait for traces to propagate
sleep 30

# Query p99 latency for the new deployment
P99=$(gcloud trace traces list \
    --project=my-staging-project \
    --start-time="$(date -u -d '5 minutes ago' +%Y-%m-%dT%H:%M:%SZ)" \
    --filter="span.attributes.deployment_sha=$COMMIT_SHA" \
    --format="value(span.duration)" \
    | sort -n | awk '{all[NR]=$0} END{print all[int(NR*0.99)]}')

BASELINE=$(cat baseline_p99.txt)

if (( $(echo "$P99 > $BASELINE * 1.1" | bc -l) )); then
    echo "P99 latency increased from $BASELINE to $P99"
    exit 1
fi
echo "P99 latency within threshold: $P99"

Security and Compliance Considerations

Traces can contain sensitive data — user IDs, request payloads, or internal IP addresses. Cloud Trace encrypts data at rest and in transit by default, but you must ensure that you don't inadvertently log PII in span attributes. OpenTelemetry provides a sanitizer processor to redact sensitive attributes. In production, we have a policy: never add raw request bodies or user emails as attributes. Instead, use hashed or anonymized identifiers. Also, consider that traces are retained for 30 days by default (configurable up to 365 days). If you need longer retention, export traces to BigQuery. For compliance (e.g., SOC2, HIPAA), you may need to audit who accesses trace data. Cloud Trace integrates with Cloud Audit Logs to track read and write operations. We enable audit logs for all trace API calls and set up alerts for suspicious access patterns.

processors:
  attributes:
    actions:
      - key: user.email
        action: delete
      - key: http.request.body
        action: hash
      - key: credit_card
        action: delete
  # Alternative: use redaction processor
  redaction:
    allow_all_keys: false
    allowed_keys:
      - http.method
      - http.status_code
      - service.name

Future-Proofing: Multi-Cloud and Hybrid Tracing

While Cloud Trace is a Google Cloud service, you can use OpenTelemetry to send traces from other clouds or on-premises. The OTel Collector can export to multiple backends simultaneously — for example, send traces to Cloud Trace for Google Cloud users and to a self-hosted Jaeger for on-prem teams. This is useful in hybrid environments. However, be aware of egress costs: sending traces from AWS to Google Cloud incurs data transfer fees. We recommend running a local collector in each environment and only exporting a sampled subset cross-cloud. Also, consider using a trace ID format that is unique across clouds — W3C TraceContext ensures compatibility. In production, we have a multi-cloud setup where our main observability backend is Cloud Trace, but we also export a 1% sample to a secondary backend for redundancy. This gives us a fallback if Cloud Trace has an outage.

exporters:
  googlecloud:
    project: my-gcp-project
  otlp:
    endpoint: jaeger.example.com:4317
    tls:
      insecure: false
      ca_file: /etc/certs/ca.crt

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [googlecloud, otlp]