Skip to content
Home C# / .NET gRPC ASP.NET Core — Silent Leak from Missing Deadline

gRPC ASP.NET Core — Silent Leak from Missing Deadline

Where developers are forged. · Structured learning · Free forever.
📍 Part of: ASP.NET → Topic 10 of 14
Without a deadline, gRPC calls hold connections indefinitely, exhausting the client pool after a few hours — leading to DEADLINE_EXCEEDED errors.
🔥 Advanced — solid C# / .NET foundation required
In this tutorial, you'll learn
Without a deadline, gRPC calls hold connections indefinitely, exhausting the client pool after a few hours — leading to DEADLINE_EXCEEDED errors.
  • gRPC is a high-performance RPC framework with strong contracts and binary serialization.
  • ASP.NET Core offers first-class support for gRPC including DI, interceptors, and health checks.
  • Always set deadlines and use cancellation tokens to prevent resource leaks.
✦ Plain-English analogy ✦ Real code with output ✦ Interview questions
Quick Answer
  • gRPC uses Protocol Buffers for binary, strongly-typed contracts
  • ASP.NET Core adds first-class gRPC support via Grpc.AspNetCore
  • Unary, server-streaming, client-streaming, and bidirectional streaming are the four RPC types
  • Interceptors act like middleware — they inspect and modify calls at the server or client
  • Production traps: deadlines not set, load balancers that don't support HTTP/2, and browser limitations
  • Performance: Protobuf serialization is 3–10x faster and 3–10x smaller than JSON
🚨 START HERE

gRPC Debugging Commands

Quick commands to diagnose gRPC issues in ASP.NET Core production environments
🟡

Deadline exceeded errors

Immediate ActionCheck if client set deadline and server has cancellation token
Commands
dotnet-counters monitor --process-id $(pid) Grpc.AspNetCore.Server.TotalCalls
kubectl logs -l app=my-grpc-service --tail=100 | grep -i deadline
Fix NowAdd deadline to client call: new GrpcCallOptions { Deadline = DateTime.UtcNow.AddSeconds(5) }
🟡

Connection refused / UNAVAILABLE

Immediate ActionTest if the port is listening and HTTP/2 is enabled
Commands
curl -v --http2-prior-knowledge http://localhost:5001/grpc.health.v1.Health/Check 2>&1 | grep -i 'HTTP/2|error'
kubectl exec -it pod-name -- wget -qO- http://localhost:5001/grpc.health.v1.Health/Check
Fix NowEnsure Kestrel is configured for HTTP/2: `builder.WebHost.ConfigureKestrel(options => options.ListenAnyIP(5001, o => o.Protocols = HttpProtocols.Http2));`
🟡

Streaming call hangs indefinitely

Immediate ActionCheck if the stream is being written or read correctly on both sides
Commands
dotnet-dump collect --process-id $(pid) && dotnet-dump analyze dump.dmp > threads.txt
grep -c 'Grpc.Core.AsyncClientStreamingCall\|WriteAsync\|MoveNext' threads.txt
Fix NowEnsure server writes complete the stream and client loops on MoveNext until cancelled.
Production Incident

Deadline Not Propagated — The Silent Connection Leak

A production gRPC service started timing out after a deployment. Clients saw increasing latency until connections were exhausted.
SymptomgRPC calls started failing with DEADLINE_EXCEEDED after a few hours. The server had plenty of CPU/memory, but client-side connection pool was exhausted.
AssumptionThe new streaming endpoint was causing more load. Adding more server replicas would fix it.
Root causeThe client had not set a deadline on the gRPC call. The server's default deadline was infinite, so long-running calls held connections open indefinitely. The client code did var call = client.SomeMethod(request); without a cancellation token or deadline.
FixAlways set a deadline on every gRPC call: var call = client.SomeMethod(request, deadline: DateTime.UtcNow.AddSeconds(5)); Use CancellationToken for cancellation. On the server side, enforce a maximum deadline via CallContext.Deadline or middleware.
Key Lesson
Always set a deadline on every gRPC call, even local ones.Test connection recovery by killing the server and observing client behaviour.Monitor gRPC metrics: pending calls, call duration, and connection count.
Production Debug Guide

Common symptoms and their root causes when gRPC calls fail in ASP.NET Core

Call returns DEADLINE_EXCEEDED after a few secondsCheck if the client set a deadline. Server might be slow — profile the server-side method. Use grpc.invoke.latency metrics.
UNAVAILABLE — No healthy upstreamLoad balancer may not support HTTP/2. Check that your reverse proxy (nginx, Envoy) is configured for HTTP/2. Verify the gRPC health check endpoint is responding.
INTERNAL error: StatusCode.UnknownEnable gRPC detailed error logging: app.UseGrpcWeb(); or set GrpcChannelOptions.ThrowOperationCanceledOnCancellation = true. Check server-side exception logs.
Call hangs for over a minute then times outLikely no deadline set or too long. Check GrpcChannel.MaxRetryBufferSize and ensure deadlines are propagated. Use CallContext.Deadline to enforce server-side timeout.

Microservices are eating the enterprise, and with that comes a brutal problem: how do dozens of services talk to each other efficiently, reliably, and without turning into a web of hand-rolled HTTP clients? REST and JSON got us far, but they carry hidden costs — verbose payloads, no enforced contracts, no built-in streaming, and type information that evaporates at the boundary. At scale, these costs compound. A single inter-service call that serializes a 50-field object as JSON a million times a day quietly burns CPU, bandwidth, and developer sanity.

gRPC was built by Google to solve exactly this. It puts a strongly-typed contract — the .proto file — at the center of every conversation, uses Protocol Buffers for binary serialization that's 3–10x smaller than equivalent JSON, and rides HTTP/2 so you get multiplexing, header compression, and real bidirectional streaming for free. ASP.NET Core embraced gRPC as a first-class citizen starting with .NET 3.0, meaning you get Kestrel's raw performance, the full DI ecosystem, middleware, and health checks all wired up without ceremony.

By the end of this article you'll know how Protobuf encodes data on the wire, how to build all four gRPC communication patterns in ASP.NET Core (unary, server-streaming, client-streaming, and bidirectional), how to write interceptors that behave like middleware, and exactly which production traps — deadlines, load balancing, browser compatibility — will catch you off-guard if you don't know they're there.

What is gRPC and Why ASP.NET Core?

gRPC is a high-performance RPC framework by Google that uses Protocol Buffers for serialization and HTTP/2 for transport. ASP.NET Core provides built-in support via the Grpc.AspNetCore package, which integrates with Kestrel's HTTP/2 pipeline. Unlike REST, where you craft endpoints manually, gRPC generates client and server stubs from a shared .proto contract. This means both sides speak the same typed language — no guessing field names, no parsing JSON strings for every call. ASP.NET Core adds dependency injection, logging, and middleware (interceptors) out of the box, making it a natural home for gRPC in the .NET ecosystem.

greet.proto · PROTOBUF
1234567891011121314151617
syntax = "proto3";

option csharp_namespace = "io.thecodeforge.Greet";

package greet;

service Greeter {
  rpc SayHello (HelloRequest) returns (HelloReply);
}

message HelloRequest {
  string name = 1;
}

message HelloReply {
  string message = 1;
}
Mental Model
Analogies for gRPC vs REST
Think of gRPC as an assembly line with blueprints, REST as a shipping dock with handwritten notes.
  • REST: each request carries its own schema (JSON). The server parses and validates every time.
  • gRPC: the schema (proto) is agreed before the first call. Serialization is a binary protocol — fast and deterministic.
  • This difference matters when you have 1000+ services exchanging billions of messages per day.
📊 Production Insight
In one production incident, a team switched from REST to gRPC for their internal inventory service. Latency dropped by 45% and CPU utilization by 30% because they eliminated JSON serialisation overhead. The catch: they had to retrain the deployment team to handle HTTP/2 load balancers.
Rule: measure the serialization cost before migrating — if your messages are small and rare, gRPC may not be worth the complexity.
🎯 Key Takeaway
gRPC enforces a typed contract; ASP.NET Core makes it a first-class citizen.
If you need performance and type safety across service boundaries, start with gRPC.

Protobuf Wire Format and Code Generation

Protocol Buffers (Protobuf) are the serialization engine of gRPC. Messages are defined in .proto files, then compiled into C# classes by the protoc compiler or the Grpc.Tools NuGet package. The on-the-wire format is a compact binary representation: field tags, wire types, and values. Varint encoding keeps small integers small, and fields can be omitted entirely if not set. This results in payloads often 3–10x smaller than equivalent JSON.

The code generator produces both the message classes (with Equals/GetHashCode overrides, ToString, and Clone) and the service stubs (client proxy and server base). In ASP.NET Core, you reference the generated files from your .csproj and implement the server base class. The generated client handles serialization, HTTP/2 framing, and connection pooling.

GreeterService.cs · C#
12345678910111213
using Grpc.Core;
using io.thecodeforge.Greet;

namespace io.thecodeforge.Services;

public class GreeterService : Greeter.GreeterBase
{
    private readonly ILogger<GreeterService> _logger;
    public GreeterService(ILogger<GreeterService> logger) => _logger = logger;

    public override Task<HelloReply> SayHello(HelloRequest request, ServerCallContext context)
    {\n        _logger.LogInformation(\"Saying hello to {Name}\", request.Name);\n        return Task.FromResult(new HelloReply\n        {\n            Message = $\"Hello {request.Name}\"\n        });\n    }\n}"
      }

Building All Four gRPC Streaming Patterns in ASP.NET Core

gRPC defines four RPC types: unary (one request, one response), server-streaming (one request, multiple responses), client-streaming (multiple requests, one response), and bidirectional streaming (multiple requests, multiple responses). ASP.NET Core supports all four with simple async method signatures.

Unary is the most common; streaming is where gRPC shines for real-time data. Server-streaming is ideal for watch operations (e.g., monitoring stream). Client-streaming is useful for uploads or batch processing. Bidirectional streaming enables chat, live collaboration, or game state sync.

Here's a server-streaming example: a service that returns stock price updates.

StockService.cs · C#
1234567891011121314151617181920212223242526
using Grpc.Core;
using io.thecodeforge.Stock;

namespace io.thecodeforge.Services;

public class StockService : Stock.StockBase
{
    public override async Task StreamStockPrices(
        StockRequest request,
        IServerStreamWriter<StockPrice> responseStream,
        ServerCallContext context)
    {\n        while (!context.CancellationToken.IsCancellationRequested)\n        {\n            var price = await GetCurrentPriceAsync(request.Symbol);\n            await responseStream.WriteAsync(price);\n            await Task.Delay(1000, context.CancellationToken);\n        }
    }

    private async Task<StockPrice> GetCurrentPriceAsync(string symbol)
    {
        // Simulate price fetch
        await Task.Delay(100);
        return new StockPrice
        {
            Symbol = symbol,
            Price = Random.Shared.NextDouble() * 1000,
            Timestamp = Google.Protobuf.WellKnownTypes.Timestamp.FromDateTime(DateTime.UtcNow)
        };
    }
}
⚠ Streaming and Cancellation
CancellationToken is your safety net. If the client disconnects, the token is cancelled. The server should throw an RpcException with StatusCode.Cancelled when cancelled. Not respecting cancellation leads to resource leaks.
📊 Production Insight
The most common streaming bug: server not checking cancellation between writes. If the client disconnects, the server continues writing to a broken stream, eventually causing an ObjectDisposedException. Always WriteAsync with CancellationToken and check it inside loops.
Rule: always pass cancellation token to WriteAsync and loop conditions.
🎯 Key Takeaway
Choose streaming patterns based on data flow, not comfort.
Always check cancellation — it's the number one streaming bug in production.
Choosing the Right RPC Type
IfSingle request, single response
UseUnary — simplest, fastest.
IfSingle request, stream of responses
UseServer-streaming — e.g., monitoring, logs.
IfStream of requests, single response
UseClient-streaming — e.g., file upload, batch query.
IfStream of requests, stream of responses
UseBidirectional streaming — e.g., chat, real-time collaboration.

Interceptors — gRPC Middleware in ASP.NET Core

Interceptors are the gRPC analogue of ASP.NET Core middleware. They run before and after each RPC call on both the server and client side. Use them for cross-cutting concerns like logging, authentication, rate limiting, and metrics.

Server-side interceptors implement Interceptor and override UnaryServerHandler, ServerStreamingServerHandler, etc. Client-side interceptors work similarly. They are registered via DI and can be ordered.

A common pattern is a server interceptor that validates the JWT token from the metadata and sets a user identity for the handler.

AuthInterceptor.cs · C#
123456789101112131415161718192021222324252627
using Grpc.Core;
using Grpc.Core.Interceptors;

namespace io.thecodeforge.Interceptors;

public class AuthInterceptor : Interceptor
{
    private readonly ILogger<AuthInterceptor> _logger;

    public AuthInterceptor(ILogger<AuthInterceptor> logger) => _logger = logger;

    public override async Task<TResponse> UnaryServerHandler<TRequest, TResponse>(\n        TRequest request,\n        ServerCallContext context,\n        UnaryServerMethod<TRequest, TResponse> continuation)
    {
        var authHeader = context.RequestHeaders.GetValue("authorization");
        if (string.IsNullOrEmpty(authHeader))
        {
            _logger.LogWarning("Missing authorization header");
            throw new RpcException(new Status(StatusCode.Unauthenticated, "Missing auth token"));
        }

        // Validate token — omitted for brevity
        // var user = await ValidateTokenAsync(authHeader);
        // context.UserState["User"] = user;

        return await continuation(request, context);
    }
}
Mental Model
Interceptors vs Middleware
Interceptors run inside the gRPC pipeline; middleware runs on the underlying HTTP/2 layer before the call reaches gRPC.
  • Middleware: can modify raw HTTP/2 frames, authentication, compression. Use for global HTTP concerns.
  • Interceptors: have access to the deserialized request/response objects and metadata headers. Use for gRPC-specific logic.
  • Recommendation: use middleware for authentication and rate limiting; use interceptors for business context enrichment and logging.
📊 Production Insight
A team once put a heavy authentication interceptor on every call. It validated the token against an external service, which added 50ms per call. That's fine for unary, but for bidirectional streaming, that interceptor runs only once at the start — but the external call still blocks the stream setup. If the external auth service is down, no streaming at all. The fix: use a health check endpoint for streaming and authenticate each message internally with a lightweight key.
Rule: keep interceptors lightweight for streaming; offload heavy validation to startup handshake.
🎯 Key Takeaway
Interceptors = middleware for gRPC.
Don't block streaming calls with heavyweight interceptors — validate once, then use context.

Production Gotchas: Deadlines, Load Balancing, and Browser Compatibility

gRPC relies on HTTP/2, which brings several production pitfalls.

  1. Deadlines: gRPC calls can hang forever if no deadline is set. Always set a client-side deadline (Deadline) and respect CancellationToken on the server.
  2. Load Balancing: L7 load balancers need to support HTTP/2 connection multiplexing. Many legacy balancers (e.g., AWS Classic LB) don't — they break gRPC. Use Envoy, Nginx with HTTP/2, or a gRPC-aware balancer. Also, gRPC connections are typically long-lived; round-robin load balancing across connections may not evenly distribute requests. Use client-side load balancing or a service mesh.
  3. Browser Compatibility: gRPC-Web or gRPC JSON transcoding needed for browser clients. ASP.NET Core supports gRPC-Web via app.UseGrpcWeb().
  4. Health Checks: Built-in gRPC health check protocol (grpc.health.v1.Health) enables liveness and readiness probes without custom endpoints.
Program.cs · C#
1234567891011121314
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddGrpc();
builder.Services.AddGrpcHealthChecks();

var app = builder.Build();

app.MapGrpcHealthChecksService();

// Enable gRPC-Web for browser clients
app.UseGrpcWeb(new GrpcWebOptions { DefaultEnabled = true });

app.MapGrpcService<io.thecodeforge.Services.GreeterService>();

app.Run();
⚠ gRPC-Web Not Enabled
If you host a Blazor WASM or React app that calls gRPC directly, the browser does not support full HTTP/2. You must enable gRPC-Web on the server. Without it, you get CORS errors or protocol violations.
📊 Production Insight
A team deployed a gRPC service behind an AWS Classic Load Balancer. Calls succeeded in dev (direct) but failed in prod with UNAVAILABLE. Root cause: Classic LB does not support HTTP/2. They switched to an Application Load Balancer with HTTPS listener and the issue resolved. Then they discovered that ALB's idle timeout (60s) was causing long-running streaming calls to disconnect. They set the idle timeout to 400s and added keepalive pings.
Rule: use gRPC-aware load balancers (Envoy, ALB, Nginx with http2) and align idle timeouts with your streaming duration.
🎯 Key Takeaway
Deadlines, load balancer HTTP/2 support, and browser compatibility are the top three production gotchas.
Test behind your actual infrastructure before going live.
Production Checklist
IfClient is a browser
UseEnable gRPC-Web or use JSON transcoding.
IfBehind a load balancer
UseEnsure LB supports HTTP/2 and configure idle timeout > streaming max duration.
IfNeed to monitor service health
UseImplement gRPC health checks and probe them from orchestrator.
🗂 gRPC vs REST vs SignalR in ASP.NET Core
Choosing the right communication paradigm for your microservices
FeaturegRPCREST/JSONSignalR
Contract enforcementStrong (via .proto file)Weak (openAPI optional)Weak (hub method signatures)
Payload sizeBinary, 3-10x smaller than JSONVerbose JSONJSON (configurable)
StreamingNative bidirectionalNot built-inBidirectional via WebSocket
PerformanceHighest (binary + HTTP/2)Moderate (HTTP/1.1 or 2)High (WebSocket persistent)
Browser supportgRPC-Web requiredNativeNative via JS client
Tooling & ecosystem.NET, Go, Java, Python, etc.Universal.NET first-class

🎯 Key Takeaways

  • gRPC is a high-performance RPC framework with strong contracts and binary serialization.
  • ASP.NET Core offers first-class support for gRPC including DI, interceptors, and health checks.
  • Always set deadlines and use cancellation tokens to prevent resource leaks.
  • Streaming requires careful cancellation handling to avoid disposal exceptions.
  • Production deployment must account for HTTP/2 load balancing and browser gRPC-Web.
  • Interceptors are powerful but can become blocking bottlenecks — keep them lightweight for streaming.

⚠ Common Mistakes to Avoid

    Not setting a deadline on gRPC calls
    Symptom

    Calls hang indefinitely, connections leak, and the service becomes unresponsive after enough idle calls accumulate.

    Fix

    Always set Deadline (e.g., DateTime.UtcNow.AddSeconds(5)) on each call. Use CancellationToken for cancellation.

    Using stream without cancellation
    Symptom

    Server continues writing after client disconnects, causing ObjectDisposedException and log noise.

    Fix

    Check context.CancellationToken in streaming loops and pass it to WriteAsync. Throw RpcException on cancellation.

    Deploying behind an HTTP/1.1 load balancer
    Symptom

    UNAVAILABLE status code, clients cannot connect. The LB terminates HTTP/2 and the gRPC protocol fails.

    Fix

    Use an L7 LB that supports HTTP/2 (AWS ALB, Nginx with http2, Envoy). Configure HttpProtocols.Http2 in Kestrel.

    Ignoring gRPC-Web for browser clients
    Symptom

    CORS errors or protocol violations when calling gRPC from JavaScript.

    Fix

    Enable app.UseGrpcWeb() and annotate services with .EnableGrpcWeb(). Use the grpc-web JavaScript client.

Interview Questions on This Topic

  • QExplain the difference between gRPC unary and streaming calls. When would you use each?Mid-levelReveal
    Unary: one request, one response. Use for standard query/response (e.g., GetUser). Server-streaming: one request, multiple responses (e.g., watch updates). Client-streaming: multiple requests, one response (e.g., batch upload). Bidirectional: both streams (e.g., chat). Choose based on the data flow pattern.
  • QHow does gRPC handle load balancing? Why is it different from REST load balancing?SeniorReveal
    gRPC uses persistent HTTP/2 connections. Traditional round-robin per connection may not balance requests well because many requests multiplex over one connection. gRPC client-side load balancing (e.g., using Grpc.Net.Client with LoadBalancingConfig) distributes calls across multiple connections or endpoints. At the server side, you need a load balancer that supports HTTP/2 (e.g., Envoy, ALB) and can route based on gRPC service/method. REST is simpler because each request usually opens a new TCP connection, so round-robin at the LB works.
  • QWhat are interceptors in gRPC and how do you implement them in ASP.NET Core?Mid-levelReveal
    Interceptors are gRPC middleware that can inspect or modify requests/responses. Implement Grpc.Core.Interceptors.Interceptor and override methods like UnaryServerHandler. Register with DI: services.AddGrpc(options => options.Interceptors.Add<MyInterceptor>()). Use for logging, auth, validation, metrics.
  • QHow do you handle deadlines and cancellation in gRPC?JuniorReveal
    Set a Deadline on the client call options (e.g., new GrpcCallOptions { Deadline = DateTime.UtcNow.AddSeconds(5) }). The server receives CancellationToken from ServerCallContext. Check it in loops and pass to async I/O. If cancelled, throw new RpcException(new Status(StatusCode.Cancelled, "...")). Without deadlines, calls may hang forever.

Frequently Asked Questions

What is gRPC with ASP.NET Core in simple terms?

gRPC with ASP.NET Core is a way to build fast, typed APIs between services using Protocol Buffers instead of JSON. ASP.NET Core provides the server and client libraries to make it easy.

Do I need a separate proto compiler?

No. The Grpc.Tools NuGet package automatically compiles .proto files during build. Add <Protobuf Include="..." /> in your .csproj.

Can I call gRPC from a browser?

Yes, with gRPC-Web. ASP.NET Core has built-in gRPC-Web support. Alternatively, use JSON transcoding to expose REST endpoints.

Why is my gRPC call failing with UNAVAILABLE?

Likely the load balancer does not support HTTP/2, or the server is not listening on an HTTP/2 port. Check Kestrel configuration and the reverse proxy.

🔥
Naren Founder & Author

Developer and founder of TheCodeForge. I built this site because I was tired of tutorials that explain what to type without explaining why it works. Every article here is written to make concepts actually click.

← PreviousMinimal APIs in ASP.NET CoreNext →Health Checks in ASP.NET Core
Forged with 🔥 at TheCodeForge.io — Where Developers Are Forged