GCP Cloud NAT: Outbound Connectivity for Private Instances

You've locked down your VPC, removed public IPs from all instances, and patted yourself on the back for a job well done. Then your CI/CD pipeline fails because apt-get update can't reach the Debian mirrors. Your private GKE nodes can't pull container images from gcr.io. Your application can't send metrics to an external monitoring service. This is the moment you realize: security without outbound connectivity is a production outage waiting to happen. GCP Cloud NAT is the production-grade solution that gives your private instances internet access without exposing them to inbound traffic. It's not a VPN, not a proxy, and definitely not a bastion host with a public IP. It's a managed NAT gateway that scales with your workloads and integrates seamlessly with Cloud Router. In this guide, I'll show you how to configure it, optimize it, and avoid the pitfalls that have taken down production environments.

What Cloud NAT Actually Does (and Doesn't Do)

Cloud NAT is a regional, managed service that performs source network address translation (SNAT) for private instances in your VPC. When a private instance sends a packet to an external IP, Cloud NAT replaces the source IP (the instance's internal IP) with one of its own external IP addresses. The destination sees traffic coming from the NAT IP, and response packets are routed back to the NAT gateway, which then forwards them to the original instance. This is fundamentally different from a proxy or VPN: Cloud NAT operates at the network layer (Layer 3/4), not the application layer. It doesn't terminate connections, inspect traffic, or require any client configuration. It also doesn't support inbound connections—no port forwarding, no load balancing. If you need inbound access, you need a different solution like Cloud Load Balancing or Identity-Aware Proxy. Cloud NAT is strictly for outbound connectivity. It works with any protocol that uses IP, including TCP, UDP, ICMP, and even GRE tunnels. However, it does not support protocols that embed IP addresses in the payload (like FTP active mode) without additional configuration. In production, you'll typically use Cloud NAT for: OS package updates, container image pulls, API calls to external services, sending logs or metrics, and license validation. It's not designed for high-throughput bulk data transfer—for that, consider Dedicated Interconnect or Partner Interconnect.

#!/bin/bash
# Create a Cloud Router (required for Cloud NAT)
gcloud compute routers create nat-router \
    --network=default \
    --region=us-central1

# Create Cloud NAT with manual IP allocation
gcloud compute routers nats create nat-config \
    --router=nat-router \
    --region=us-central1 \
    --nat-external-ip-pool=IP_ADDRESS_1,IP_ADDRESS_2 \
    --nat-all-subnet-ip-ranges \
    --enable-logging \
    --log-filter=ERRORS_ONLY

Manual vs. Automatic NAT IP Allocation

When you create a Cloud NAT, you must decide how to allocate external IP addresses. You have two options: automatic and manual. Automatic NAT IP assignment lets GCP allocate IP addresses from a pool of regional IPs. This is the simplest option—you don't need to manage IP addresses, and GCP handles scaling. However, you lose control over which IPs are used, which can be problematic if external services whitelist your IPs. Manual NAT IP assignment requires you to reserve and assign specific external IP addresses. This is essential for production environments where you need predictable IPs for firewall rules, API rate limiting, or compliance. You can assign up to 32 IPs per NAT gateway. The number of IPs you need depends on the number of concurrent connections. Each NAT IP supports approximately 64,000 source ports. If you have many instances making many connections, you may exhaust ports and experience connection failures. A good rule of thumb: start with 2-4 IPs for small deployments, and monitor port usage via Cloud Monitoring. You can add IPs later without downtime. In production, always use manual IP assignment with reserved static IPs. This ensures your IPs don't change if the NAT gateway is recreated, and you can whitelist them in external systems.

#!/bin/bash
# Reserve two static external IP addresses
gcloud compute addresses create nat-ip-1 \
    --region=us-central1

gcloud compute addresses create nat-ip-2 \
    --region=us-central1

# List reserved IPs
gcloud compute addresses list --filter="name~'nat-ip-*'"

Configuring NAT Rules for Subnets and VMs

Cloud NAT allows you to specify which subnets and VMs should use the NAT gateway. You can choose from three modes: NAT all subnets in the region, NAT specific subnets, or NAT specific VMs (using tags or service accounts). The most common pattern is to NAT all subnets in a region, which is simple and ensures all private instances have outbound access. However, in production, you may want to restrict NAT to specific subnets for security or cost control. For example, you might have a subnet for internal-only services that should never have internet access. To exclude a subnet, you use the --nat-all-subnet-ip-ranges flag with --nat-custom-subnet-ip-ranges and list the subnets you want to include. For finer control, you can use VM tags or service accounts. This is useful when you have a mix of instances in the same subnet but only some need internet access. For example, you might tag instances with allow-outbound and configure NAT to only apply to those tags. Note that tags are instance-level metadata, not network tags. To use tags, you must specify --nat-custom-subnet-ip-ranges and then use --source-tags or --source-service-accounts. This approach adds complexity but gives you granular control. In practice, I recommend starting with subnet-level NAT and only moving to VM-level if you have a specific requirement.

#!/bin/bash
# Create NAT that only applies to VMs with tag 'nat-access'
gcloud compute routers nats create nat-config \
    --router=nat-router \
    --region=us-central1 \
    --nat-external-ip-pool=IP_ADDRESS_1,IP_ADDRESS_2 \
    --nat-custom-subnet-ip-ranges=subnet-a,subnet-b \
    --source-tags=nat-access

# Create a VM with the tag
gcloud compute instances create private-vm \
    --zone=us-central1-a \
    --subnet=subnet-a \
    --no-address \
    --tags=nat-access

Port Exhaustion: The Silent Killer of Outbound Connections

Cloud NAT uses source port mapping to track connections. Each NAT IP has a limited number of source ports (approximately 64,000). When an instance makes an outbound connection, Cloud NAT assigns a source port from the NAT IP's pool. If all ports are in use, new connections fail. This is called port exhaustion, and it's one of the most common production issues with Cloud NAT. Symptoms include: intermittent connection timeouts, failed API calls, and Cannot assign requested address errors in application logs. Port exhaustion typically happens when: you have many instances behind a single NAT IP, instances make many short-lived connections (e.g., HTTP keep-alive disabled), or connections are not properly closed (e.g., connection leaks). To diagnose port exhaustion, enable NAT logging and look for NO_MAPPING or PORT_EXHAUSTION log entries. You can also monitor the nat/port_usage metric in Cloud Monitoring. To mitigate port exhaustion: increase the number of NAT IPs (each IP adds ~64K ports), enable endpoint-independent mapping (which allows port reuse for different destinations), or reduce the number of concurrent connections from your instances. In production, I recommend starting with 4 NAT IPs and monitoring port usage. Set up an alert when port usage exceeds 80%. Also, consider using connection pooling or HTTP keep-alive in your applications to reduce the number of concurrent connections.

#!/bin/bash
# Create a Cloud Monitoring alert for port usage
gcloud alpha monitoring policies create \
    --display-name="NAT Port Usage Alert" \
    --condition-display-name="Port usage > 80%" \
    --condition-filter='resource.type = "nat_gateway" AND metric.type = "networking.googleapis.com/nat/port_usage"' \
    --condition-threshold-value=0.8 \
    --condition-threshold-duration=300s \
    --notification-channels=YOUR_CHANNEL_ID

Endpoint-Independent Mapping and Filtering

Cloud NAT supports two types of mapping and filtering: endpoint-independent and address-dependent. By default, Cloud NAT uses address-dependent mapping, which means that a source port is reused only for the same destination IP and port. This is more secure but can lead to port exhaustion under heavy load. Endpoint-independent mapping allows a source port to be reused for different destinations, increasing the effective number of connections per NAT IP. However, it also reduces security because an external host could potentially send traffic to the NAT IP and have it forwarded to an internal instance if the source port matches an existing mapping. To mitigate this, Cloud NAT also supports endpoint-independent filtering, which drops unsolicited inbound packets. In production, I recommend enabling endpoint-independent mapping with endpoint-independent filtering. This gives you the best performance without sacrificing security. To enable this, you must set the --nat-all-subnet-ip-ranges flag and use the --enable-endpoint-independent-mapping flag when creating or updating the NAT configuration. Note that this feature is not available in all regions; check the GCP documentation for regional availability. Also, endpoint-independent mapping changes the behavior of the NAT gateway, so test thoroughly in a non-production environment first.

#!/bin/bash
# Update existing NAT to use endpoint-independent mapping
gcloud compute routers nats update nat-config \
    --router=nat-router \
    --region=us-central1 \
    --enable-endpoint-independent-mapping

# Verify the configuration
gcloud compute routers nats describe nat-config \
    --router=nat-router \
    --region=us-central1

Idle Timeout Configuration and Connection Draining

Cloud NAT has configurable idle timeouts for TCP, UDP, and ICMP connections. If a connection is idle for longer than the timeout, the NAT mapping is removed, and the source port becomes available for reuse. The default timeouts are: TCP established: 1200 seconds (20 minutes), TCP transitory: 30 seconds, UDP: 30 seconds, ICMP: 30 seconds. In production, you may need to adjust these timeouts based on your application's behavior. For example, if your application uses long-lived TCP connections (e.g., database connections, WebSockets), you should increase the TCP established timeout to prevent premature disconnection. Conversely, if you have many short-lived connections, you can decrease the timeout to free up ports faster. To change timeouts, use the --tcp-established-idle-timeout, --tcp-transitory-idle-timeout, --udp-idle-timeout, and --icmp-idle-timeout flags. Be careful: setting timeouts too low can cause active connections to be dropped, leading to application errors. Setting them too high can delay port reuse and contribute to port exhaustion. In practice, I recommend keeping the defaults unless you have a specific reason to change them. If you do change them, monitor the impact on both connection stability and port usage. Also, note that Cloud NAT does not support connection draining. When you delete or update a NAT gateway, existing connections are immediately terminated. Plan maintenance windows accordingly.

#!/bin/bash
# Update NAT with custom idle timeouts
gcloud compute routers nats update nat-config \
    --router=nat-router \
    --region=us-central1 \
    --tcp-established-idle-timeout=3600 \
    --tcp-transitory-idle-timeout=60 \
    --udp-idle-timeout=60 \
    --icmp-idle-timeout=30

Logging and Monitoring Cloud NAT

Cloud NAT supports two types of logging: NAT logs and VPC flow logs. NAT logs record information about each NAT translation, including source IP, destination IP, source port, and action (e.g., MAPPING_CREATED, MAPPING_DELETED, NO_MAPPING). You can enable NAT logs with different filters: ERRORS_ONLY, TRANSLATIONS_ONLY, or ALL. In production, I recommend starting with ERRORS_ONLY to capture failures without overwhelming your logging system. If you need to debug connectivity issues, you can temporarily switch to ALL. NAT logs are written to Cloud Logging and can be exported to BigQuery for analysis. VPC flow logs, on the other hand, capture information about all IP traffic in your VPC, including traffic that goes through NAT. They are useful for network troubleshooting and security analysis. However, VPC flow logs can be expensive and generate a lot of data. Use them selectively. For monitoring, Cloud NAT exposes several metrics in Cloud Monitoring: nat/port_usage, nat/connections, nat/packets_sent, nat/bytes_sent, etc. I recommend creating a dashboard that shows port usage, connection counts, and error rates. Set up alerts for port usage > 80%, connection failures, and any NO_MAPPING log entries. Also, consider using the nat/connections metric to detect sudden drops that might indicate a NAT gateway failure.

#!/bin/bash
# Enable NAT logging with ERRORS_ONLY filter
gcloud compute routers nats update nat-config \
    --router=nat-router \
    --region=us-central1 \
    --enable-logging \
    --log-filter=ERRORS_ONLY

# View recent NAT logs
gcloud logging read "resource.type=nat_gateway AND severity>=ERROR" \
    --limit=10 \
    --format=json

Static vs. Dynamic Port Allocation

Cloud NAT supports two port allocation methods: static and dynamic. With static port allocation (default for Public NAT), each VM gets a fixed minimum number of ports (default 64). This is simple but wasteful if VMs have uneven egress usage—a VM that needs 1000 ports gets only 64, while an idle VM also gets 64 that it never uses. With dynamic port allocation (default for Private NAT), each VM starts with a minimum number of ports (default 32) and can burst up to a configurable maximum. When a VM approaches exhaustion, its allocation is doubled. When usage drops, ports are deallocated and returned to the pool. This is far more efficient for heterogeneous workloads. However, dynamic allocation breaks existing connections when switching methods, and it cannot be used with endpoint-independent mapping. Choose static allocation for predictable, uniform workloads and dynamic allocation for variable or mixed workloads. In production, monitor port usage over time to determine which method fits. If you see uneven port consumption across VMs, switch to dynamic.

# Update NAT to use dynamic port allocation
gcloud compute routers nats update nat-config \
    --router=nat-router \
    --region=us-central1 \
    --nat-all-subnet-ip-ranges \
    --dynamic-port-allocation \
    --min-ports-per-vm=32 \
    --max-ports-per-vm=512

# Verify the configuration
gcloud compute routers nats describe nat-config \
    --router=nat-router \
    --region=us-central1 \
    --format='get(enableDynamicPortAllocation, minPortsPerVm, maxPortsPerVm)'

Private NAT: VPC-to-VPC and Hybrid NAT

Private NAT is a separate NAT type that translates private IP addresses to other private IP addresses, rather than to public internet IPs. It enables communication between VPC networks with overlapping IP ranges, between VPC and on-premises networks via Cloud Interconnect or Cloud VPN, and between VPC networks connected through Network Connectivity Center (NCC) spokes. Unlike Public NAT, Private NAT uses IP addresses from a dedicated Private NAT subnet (created with --purpose=PRIVATE_NAT). You cannot automatically assign IPs—you manually specify the subnet. Private NAT uses dynamic port allocation by default and does not support endpoint-independent mapping. Common use cases: connecting two merged organizations with overlapping VPC ranges, routing traffic from on-premises to Google Cloud without IP conflicts, and enabling communication between NCC-attached VPCs. Private NAT is billed separately from Public NAT, and you can run both on the same subnet for different purposes.

# Create a Private NAT subnet
gcloud compute networks subnets create private-nat-subnet \
    --network=default \
    --region=us-central1 \
    --range=10.0.0.0/24 \
    --purpose=PRIVATE_NAT

# Create a Cloud Router for Private NAT
gcloud compute routers create private-nat-router \
    --network=default \
    --region=us-central1

# Create Private NAT gateway
gcloud compute routers nats create private-nat-config \
    --router=private-nat-router \
    --region=us-central1 \
    --nat-all-subnet-ip-ranges \
    --nat-custom-subnet-ip-ranges=private-nat-subnet \
    --type=PRIVATE

High Availability and Multi-Region Considerations

Cloud NAT is a regional service. If the region goes down, your NAT gateway goes with it. For high availability, you need to deploy Cloud NAT in multiple regions and route traffic accordingly. However, Cloud NAT does not support cross-region failover automatically. You must design your application to handle regional failures. One approach is to use a global load balancer in front of your instances and have instances in multiple regions. If one region fails, the load balancer routes traffic to healthy regions. Another approach is to use Cloud DNS with health checks to direct traffic to a backup region. For the NAT gateway itself, you can create a secondary NAT gateway in a different region and configure your instances to use it via routing. However, this is complex and rarely needed. In practice, most applications can tolerate a regional NAT outage because they don't require constant outbound connectivity. If you need high availability, consider using a third-party NAT appliance or a proxy fleet. Also, note that Cloud NAT is designed to be resilient within a region. It uses multiple VMs in a managed instance group, so a single VM failure does not cause an outage. However, a regional failure will take down the entire NAT gateway. For multi-region deployments, ensure that your instances can fall back to a different NAT gateway or have a mechanism to route traffic through a VPN to another region.

#!/bin/bash
# Create NAT in us-central1 and europe-west1
gcloud compute routers create nat-router-us \
    --network=default \
    --region=us-central1

gcloud compute routers nats create nat-config-us \
    --router=nat-router-us \
    --region=us-central1 \
    --nat-external-ip-pool=NAT_IP_US \
    --nat-all-subnet-ip-ranges

gcloud compute routers create nat-router-eu \
    --network=default \
    --region=europe-west1

gcloud compute routers nats create nat-config-eu \
    --router=nat-router-eu \
    --region=europe-west1 \
    --nat-external-ip-pool=NAT_IP_EU \
    --nat-all-subnet-ip-ranges

Cost Optimization and Budgeting

Cloud NAT pricing consists of two components: a per-hour charge for the NAT gateway itself (approximately $0.045 per hour per gateway) and a per-GB charge for data processed (approximately $0.045 per GB). Additionally, you pay for the external IP addresses if you use manual IPs (static IPs have a small hourly charge when not in use, but no charge when attached to a NAT gateway). For high-throughput workloads, the data processing cost can dominate. To optimize costs: use automatic IP assignment to avoid static IP charges (but be aware of the stability trade-off), minimize the number of NAT gateways (one per region is usually sufficient), and consider using Private Google Access for Google APIs and services to avoid NAT data processing charges. Private Google Access allows instances with only internal IPs to reach Google APIs and services (like Cloud Storage, BigQuery) without going through NAT. This is free and should be enabled by default. Also, monitor your NAT data processing volume and set budgets or alerts to avoid surprises. In production, I recommend enabling Private Google Access on all subnets and using NAT only for non-Google external destinations. This can reduce your NAT bill by 50% or more. Finally, consider using committed use discounts for your VMs to offset some costs, but note that NAT itself does not have committed use discounts.

#!/bin/bash
# Enable Private Google Access on a subnet
gcloud compute networks subnets update subnet-a \
    --region=us-central1 \
    --enable-private-ip-google-access

# Verify
gcloud compute networks subnets describe subnet-a \
    --region=us-central1 \
    --format='get(privateIpGoogleAccess)'

Troubleshooting Common Cloud NAT Issues

Despite its simplicity, Cloud NAT can fail in subtle ways. Here are the most common issues and how to fix them. 1. No outbound connectivity: First, verify that the instance has no public IP (gcloud compute instances describe INSTANCE --format='get(networkInterfaces[0].accessConfigs)' should be empty). Then check that the subnet is included in the NAT configuration. Use gcloud compute routers nats describe to see the configured subnets. Also, ensure the Cloud Router is healthy. 2. Intermittent connectivity: This is often port exhaustion. Check NAT logs for NO_MAPPING errors and monitor port usage. Add more NAT IPs or enable endpoint-independent mapping. 3. Specific destinations unreachable: Some external services block traffic from known cloud IP ranges. Use manual IPs and whitelist them. Also, check if the destination requires a specific source port or protocol. 4. High latency: NAT adds minimal latency (usually <1ms), but if you see high latency, check if the NAT gateway is in the same region as your instances. Cross-region NAT adds significant latency. 5. Connection resets: This can happen if idle timeouts are too low. Increase the TCP established timeout. Also, check if the application is sending keep-alive packets. 6. NAT gateway not working after update: If you update the NAT configuration, existing connections are dropped. Plan for this. For persistent issues, enable NAT logs with ALL filter and analyze the logs in Cloud Logging. You can also use gcloud compute routers nats get-status to see the current state of the NAT gateway.

#!/bin/bash
# Check NAT status
gcloud compute routers nats get-status nat-config \
    --router=nat-router \
    --region=us-central1

# Test connectivity from a VM
gcloud compute ssh private-vm --zone=us-central1-a --command="curl -v https://example.com"

Alternatives to Cloud NAT: When Not to Use It

Cloud NAT is not always the right solution. Here are alternatives and when to use them. 1. Public IPs: If you have a small number of instances that need outbound connectivity, assigning a public IP might be simpler and cheaper. However, this increases the attack surface. Use only for non-production or low-security workloads. 2. Cloud VPN or Interconnect: If you need to route traffic through an on-premises network (e.g., for compliance), use Cloud VPN or Dedicated Interconnect. NAT cannot replace a VPN. 3. Private Service Connect: For accessing Google APIs and services, Private Google Access is better than NAT. For third-party services that support Private Service Connect, use that instead. 4. Proxy servers: If you need application-layer filtering, caching, or authentication, use a proxy (e.g., Squid, HAProxy) instead of NAT. NAT operates at the network layer and cannot inspect traffic. 5. Cloud Armor: If you need to protect against DDoS attacks, Cloud Armor works with Cloud Load Balancing, not NAT. NAT does not provide any security beyond hiding internal IPs. 6. Serverless VPC Access: For Cloud Functions or Cloud Run, use Serverless VPC Access to connect to your VPC, and then use NAT for outbound traffic. However, Serverless VPC Access has its own limitations. In summary, use Cloud NAT when you need simple, scalable outbound connectivity for private VMs without inbound access. For anything more complex, consider the alternatives.

#!/bin/bash
# This is a conceptual comparison, not runnable code
echo "Cloud NAT: Outbound-only, no inbound, no app-layer filtering"
echo "Public IP: Simple but insecure"
echo "Cloud VPN: Site-to-site, encrypted, for hybrid cloud"
echo "Proxy: App-layer control, higher latency"

Production Checklist for Cloud NAT Deployment

Before deploying Cloud NAT to production, run through this checklist. 1. Reserve static IPs: Use manual IP allocation with reserved static IPs. Whitelist these IPs with any external services. 2. Enable logging: Enable NAT logs with ERRORS_ONLY filter at minimum. Set up a Cloud Logging sink to export logs to BigQuery for analysis. 3. Set up monitoring: Create a Cloud Monitoring dashboard with port usage, connection counts, and error rates. Set alerts for port usage > 80% and any NO_MAPPING errors. 4. Configure idle timeouts: Adjust timeouts based on your application's connection patterns. Test in staging first. 5. Enable Private Google Access: On all subnets to reduce NAT traffic and costs. 6. Test failover: If using multiple regions, test that instances can fall back to a different NAT gateway. 7. Document IPs and configuration: Keep a record of your NAT IPs, subnets, and any whitelisting. This is critical for incident response. 8. Review firewall rules: Ensure egress firewall rules allow traffic to required destinations. Don't forget DNS (UDP 53) and NTP (UDP 123). 9. Plan for updates: Since NAT updates drop connections, schedule changes during maintenance windows. Consider using a blue/green deployment with two NAT gateways. 10. Test from a VM: After deployment, SSH into a private VM and test connectivity to several external endpoints. Verify that the source IP is one of your NAT IPs. This checklist will help you avoid common pitfalls and ensure a smooth production deployment.

#!/bin/bash
# Verify NAT IP from a VM
gcloud compute ssh private-vm --zone=us-central1-a --command="curl -s ifconfig.me"