AWS PrivateLink: VPC Endpoints and Endpoint Services

You’ve spent weeks hardening your security groups, locking down IAM policies, and encrypting everything in transit. Then someone decides to route traffic to an S3 bucket through a NAT gateway over the internet. That’s a security hole you can drive a truck through. AWS PrivateLink fixes this by keeping traffic entirely within the AWS network, never touching the public internet. It’s not just about security—it’s about operational simplicity and compliance. When you use PrivateLink, you eliminate the need for internet gateways, NAT devices, and firewall rules for inter-VPC or service access. But misconfiguring it can silently break your architecture. I’ve seen teams spend days debugging timeouts only to realize their endpoint policy was too restrictive. This article covers what PrivateLink is, how to set it up correctly, and the gotchas that will bite you in production.

Why PrivateLink? The Problem with VPC Peering and NAT

VPC peering and NAT gateways are common approaches for connecting services across VPCs, but they come with significant operational baggage. VPC peering requires transitive routing hacks, non-overlapping CIDRs, and manual route table updates that break under scale. NAT gateways introduce a single point of failure and egress costs that spiral as traffic grows. Worse, neither provides granular access control — once traffic leaves your VPC, you lose visibility into which consumer is talking to which service. PrivateLink solves this by exposing services via Elastic Network Interfaces (ENIs) inside consumer VPCs, eliminating the need for internet gateways, NAT devices, or peering connections. Traffic stays within the AWS backbone, and IAM policies can enforce per-consumer access. This is not just a networking improvement; it's a security boundary shift. You stop trusting the network and start trusting identity.

aws ec2 create-vpc-endpoint \
  --vpc-id vpc-0abcd1234 \
  --service-name com.amazonaws.vpce.us-east-1.vpce-svc-0example \
  --vpc-endpoint-type Interface \
  --subnet-ids subnet-0subnet1 subnet-0subnet2 \
  --security-group-ids sg-0sg12345 \
  --private-dns-enabled

Anatomy of a VPC Endpoint: Interface vs Gateway Endpoints

AWS offers two types of VPC endpoints: Interface endpoints (powered by PrivateLink) and Gateway endpoints (for S3 and DynamoDB). Interface endpoints create an Elastic Network Interface (ENI) in your subnet with a private IP, allowing traffic to reach supported AWS services or your own endpoint services. Gateway endpoints are a route table entry that directs traffic to S3 or DynamoDB without leaving the AWS network, but they only work for those two services and cannot be used for custom services. For PrivateLink, you always use Interface endpoints. The key difference: Interface endpoints cost $0.01 per AZ per hour plus data processing charges, while Gateway endpoints are free but limited. In practice, use Gateway endpoints for S3/DynamoDB to save costs, and Interface endpoints for everything else. Never mix them — a common mistake is trying to use a Gateway endpoint for a custom service, which silently fails.

aws ec2 describe-vpc-endpoint-services \
  --query 'ServiceDetails[?ServiceType==`Interface`].ServiceName' \
  --output table

Building an Endpoint Service: The Provider Side

To expose your application via PrivateLink, you create a Network Load Balancer (NLB) in your service VPC, then register it as an endpoint service. The NLB must be internal (no public IP) and in the same VPC as your application instances. The endpoint service is defined by a service name (e.g., com.amazonaws.vpce.us-east-1.vpce-svc-0example) and a set of allowed principals (AWS accounts or IAM roles). You can also configure acceptance required: if true, consumers must request access and you approve each connection. In production, always require acceptance to prevent unauthorized access. The NLB handles health checks and load balancing, so your application must be stateless behind it. Once the endpoint service is live, consumers can create VPC endpoints pointing to your service name. The ENIs in their VPC will route traffic to your NLB, which forwards to your application.

aws ec2 create-vpc-endpoint-service-configuration \
  --network-load-balancer-arns arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/net/my-nlb/abc123def456 \
  --acceptance-required \
  --add-private-dns-names example.com \
  --tag-specifications ResourceType=vpc-endpoint-service,Tags=[{Key=Environment,Key=Production}]

Consumer Side: Creating a VPC Endpoint

As a consumer, you create a VPC endpoint of type Interface in your VPC, specifying the service name provided by the producer. You must select subnets (one per AZ) where ENIs will be created. Each ENI gets a private IP from your subnet range. You also attach a security group that controls inbound traffic to the endpoint — this is your first line of defense. Optionally, enable private DNS so that DNS queries for the service's endpoint (e.g., my-service.example.com) resolve to the private IPs of the ENIs. Without private DNS, you must manually manage DNS or use Route 53 private hosted zones. In production, always enable private DNS to avoid brittle DNS workarounds. After creation, the endpoint state is 'pendingAcceptance' until the producer approves it. Once accepted, traffic flows.

aws ec2 create-vpc-endpoint \
  --vpc-id vpc-0consumer \
  --vpc-endpoint-type Interface \
  --service-name com.amazonaws.vpce.us-east-1.vpce-svc-0example \
  --subnet-ids subnet-0subnetA subnet-0subnetB \
  --security-group-ids sg-0consumer-sg \
  --private-dns-enabled \
  --tag-specifications ResourceType=vpc-endpoint,Tags=[{Key=Environment,Value=Production}]

Security: IAM Policies and Security Groups

PrivateLink security operates at two levels: network and identity. Network security is enforced by security groups attached to the VPC endpoint ENI. These control inbound traffic from the consumer's VPC to the endpoint. Identity security is enforced by IAM policies on the endpoint service that restrict which principals (AWS accounts, IAM roles, or users) can create endpoints to your service. Additionally, you can use AWS PrivateLink endpoint policies (for AWS services) to control what actions consumers can perform. For custom services, the NLB security group controls traffic from the endpoint ENIs to your application. In production, use a combination: security groups to restrict source IPs (if known), and IAM to restrict accounts. Never rely solely on security groups — they don't prevent a malicious account from creating an endpoint if the service allows all principals.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::123456789012:root",
          "arn:aws:iam::111111111111:role/ConsumerRole"
        ]
      },
      "Action": "ec2:CreateVpcEndpoint",
      "Resource": "arn:aws:ec2:us-east-1:123456789012:vpc-endpoint-service/vpce-svc-0example"
    }
  ]
}

DNS and Private Hosted Zones: Making It Work

PrivateLink endpoints support private DNS names. When you enable private DNS on a VPC endpoint, AWS automatically creates a Route 53 private hosted zone for the service's DNS name (e.g., my-service.example.com) and associates it with your VPC. This zone contains A records pointing to the ENI private IPs. For custom endpoint services, you must configure a private DNS name on the service side and verify domain ownership. Without private DNS, consumers must either use the endpoint-specific DNS name (which includes the VPC endpoint ID) or create their own Route 53 records. In production, always use private DNS — it's free and eliminates DNS management overhead. However, be aware that private DNS only works within the VPC; if you need cross-region or on-premises resolution, you'll need Route 53 Resolver endpoints or conditional forwarding.

nslookup my-service.example.com
# Output shows private IPs from the endpoint ENIs
Server:   10.0.0.2
Address:  10.0.0.2#53

Name: my-service.example.com
Address: 10.0.1.45
Name: my-service.example.com
Address: 10.0.2.67

Monitoring and Troubleshooting PrivateLink

Monitor PrivateLink using VPC Flow Logs, CloudWatch metrics, and endpoint service events. Key metrics: ActiveConnections, NewConnections, PacketsIn/Out, and BytesProcessed. For endpoint services, track the number of endpoints and their states. Common issues: endpoint stuck in 'pendingAcceptance' (producer hasn't approved), security group blocking traffic, or NLB health checks failing. Use VPC Flow Logs to see if traffic reaches the ENI. If you see 'REJECT' in flow logs, check security groups. If traffic reaches the ENI but not the application, check NLB target group health. Another frequent issue: cross-account DNS resolution fails if the consumer's VPC doesn't have the private hosted zone associated. In production, set up CloudWatch alarms on ActiveConnections dropping to zero — that indicates a connectivity failure.

aws ec2 describe-vpc-endpoints \
  --vpc-endpoint-ids vpce-0consumer123 \
  --query 'VpcEndpoints[0].State' \
  --output text
# Output: available

Cost Optimization: Data Processing and Cross-AZ Charges

PrivateLink costs include: hourly charges per endpoint per AZ ($0.01/hr), data processing charges ($0.01/GB for data processed by the endpoint), and cross-AZ data transfer costs if the consumer and producer are in different AZs. To optimize: use one endpoint per AZ and ensure consumers in the same AZ connect to the endpoint in that AZ (use private DNS which returns AZ-local IPs). Avoid creating endpoints in every AZ if you only need one. Also, minimize data transfer by co-locating consumer and producer in the same region and same AZ if possible. For high-throughput workloads, consider using Gateway endpoints for S3/DynamoDB to avoid data processing charges. In production, we saved 30% by consolidating endpoints and using AZ affinity.

# Example: 3 AZs, 100GB/day data processed
# Hourly: 3 endpoints * $0.01 * 24 = $0.72/day
# Data: 100GB * $0.01 = $1.00/day
# Total: ~$1.72/day or ~$51.6/month

Scaling PrivateLink: Multi-AZ and Multi-Region

For high availability, create VPC endpoints in at least two AZs. The NLB backing the endpoint service must also be multi-AZ. PrivateLink does not natively support cross-region; endpoints are region-specific. To connect across regions, use VPC peering or Transit Gateway between the consumer VPC and a proxy VPC that hosts the endpoint, or use AWS PrivateLink with inter-region VPC peering (which adds complexity). Alternatively, expose the service via a global accelerator. In production, we use a hub-and-spoke model: a central VPC hosts the endpoint service, and consumer VPCs in the same region connect via PrivateLink. For cross-region, we replicate the service or use a VPN. Never rely on a single AZ — an AZ outage will take down your connectivity.

aws ec2 create-vpc-endpoint \
  --vpc-id vpc-0consumer \
  --vpc-endpoint-type Interface \
  --service-name com.amazonaws.vpce.us-east-1.vpce-svc-0example \
  --subnet-ids subnet-0az1 subnet-0az2 subnet-0az3 \
  --security-group-ids sg-0consumer-sg \
  --private-dns-enabled

Real-World Failure Modes and Mitigations

PrivateLink failures often stem from misconfiguration. Common failure modes: (1) Security group on the endpoint ENI blocks traffic — always verify with flow logs. (2) NLB target group health checks fail — ensure your application responds to health checks on the correct port and path. (3) Endpoint service acceptance required but producer never approves — set up automation to auto-approve trusted accounts. (4) Private DNS not enabled — leads to DNS resolution failures. (5) Cross-account IAM policies missing — consumers get 'AccessDenied' when creating endpoints. Mitigations: use AWS Config rules to enforce private DNS enabled, security group rules, and acceptance required. Set up CloudWatch alarms on endpoint state changes. In production, we also use a Lambda function that auto-approves endpoints from known accounts and sends alerts for unknown ones.

import boto3
import os

ec2 = boto3.client('ec2')

def lambda_handler(event, context):
    allowed_accounts = os.environ['ALLOWED_ACCOUNTS'].split(',')
    endpoint_id = event['detail']['requestParameters']['vpcEndpointId']
    service_id = event['detail']['requestParameters']['serviceId']
    
    # Get endpoint details
    endpoint = ec2.describe_vpc_endpoints(VpcEndpointIds=[endpoint_id])
    owner_id = endpoint['VpcEndpoints'][0]['OwnerId']
    
    if owner_id in allowed_accounts:
        ec2.accept_vpc_endpoint_connections(
            ServiceId=service_id,
            VpcEndpointIds=[endpoint_id]
        )
        print(f'Auto-approved endpoint {endpoint_id} from account {owner_id}')
    else:
        print(f'Rejected endpoint {endpoint_id} from unknown account {owner_id}')

Alternatives to PrivateLink: When Not to Use It

PrivateLink is not always the best choice. For high-throughput, low-latency workloads within the same VPC, use direct VPC connectivity. For S3 and DynamoDB, use Gateway endpoints (free). For on-premises connectivity, use Direct Connect or VPN. If you need transitive routing across many VPCs, Transit Gateway is simpler than a mesh of PrivateLink endpoints. PrivateLink also has a limit of 1000 endpoints per VPC (soft limit, can be increased). If you exceed that, consider using Transit Gateway or VPC peering. In production, we use a decision matrix: same VPC? Direct connect. Same region, different VPC? PrivateLink. Cross-region? Transit Gateway or VPN. On-premises? Direct Connect. Never force PrivateLink where simpler solutions exist.

aws ec2 describe-vpc-endpoints \
  --query 'length(VpcEndpoints)' \
  --output text
# Output: 42 (current count)

Production Checklist: Deploying PrivateLink Safely

Before deploying PrivateLink in production, verify: (1) NLB is internal and multi-AZ with healthy targets. (2) Endpoint service requires acceptance and has IAM policy restricting principals. (3) Consumer VPC endpoints have private DNS enabled and security groups allowing traffic from application subnets. (4) Flow logs are enabled on both sides. (5) CloudWatch alarms on endpoint state and active connections. (6) DNS resolution works end-to-end (test with nslookup). (7) Cross-account IAM roles are correctly set up. (8) Cost estimates are within budget. (9) Have a rollback plan: if PrivateLink fails, can you fall back to a direct connection or VPN? In production, we always deploy a parallel VPN connection as a backup during the initial migration. Once stable, we remove the backup.

# Test connectivity from consumer to service
curl -v https://my-service.example.com/health
# Expected: HTTP 200 with response body
# If fails, check:
# 1. Security groups on endpoint ENI
# 2. NLB target group health
# 3. DNS resolution
# 4. Endpoint state (should be 'available')