GCP Hybrid Connectivity: Cloud VPN, Interconnect, and Partner Interconnect

We spent three days debugging a packet loss issue that turned out to be a misconfigured MTU on a Cloud VPN tunnel. The fix took 30 seconds. That's the reality of hybrid connectivity: one wrong setting can silently degrade your entire production pipeline. GCP offers three ways to connect your on-premises network—Cloud VPN, Dedicated Interconnect, and Partner Interconnect—and choosing wrong can cost you latency, bandwidth, or both. This isn't about theory; it's about which circuit to order, how to route BGP, and what to do when your VPN drops at 3 AM. If you're running anything critical across a hybrid network, you need to understand the tradeoffs between these options and how to implement them without waking up to a pager storm.

Cloud VPN: The Swiss Army Knife of Hybrid Connectivity

Cloud VPN is the simplest and most accessible hybrid connectivity option. It uses IPSec tunnels over the public internet to connect your on-premises network to your VPC. It's ideal for development environments, low-bandwidth workloads, or as a backup link for more expensive interconnects. GCP supports both route-based (dynamic) and policy-based (static) VPNs, but route-based with BGP is the only production-worthy choice. The key limitation is bandwidth: each tunnel maxes out at 3 Gbps (with HA VPN), and latency is subject to internet conditions. You also need to handle encryption overhead and MTU fragmentation. For production, always use HA VPN (two tunnels in different regions) and configure BGP for automatic failover. Never rely on a single tunnel—we've seen entire regions go dark due to a fiber cut.

resource "google_compute_ha_vpn_gateway" "ha_gateway" {
  name    = "ha-vpn-gateway"
  network = google_compute_network.vpc.id
  region  = "us-central1"
}

resource "google_compute_external_vpn_gateway" "on_prem_gateway" {
  name            = "on-prem-gateway"
  redundancy_type = "TWO_IPS_REDUNDANCY"
  interface {
    id         = 0
    ip_address = "203.0.113.1"
  }
  interface {
    id         = 1
    ip_address = "203.0.113.2"
  }
}

resource "google_compute_vpn_tunnel" "tunnel0" {
  name          = "tunnel-0"
  region        = "us-central1"
  vpn_gateway   = google_compute_ha_vpn_gateway.ha_gateway.id
  peer_gcp_gateway = google_compute_external_vpn_gateway.on_prem_gateway.id
  interface     = 0
  shared_secret = "my-secret-key"
  ike_version   = 2
}

resource "google_compute_vpn_tunnel" "tunnel1" {
  name          = "tunnel-1"
  region        = "us-central1"
  vpn_gateway   = google_compute_ha_vpn_gateway.ha_gateway.id
  peer_gcp_gateway = google_compute_external_vpn_gateway.on_prem_gateway.id
  interface     = 1
  shared_secret = "my-secret-key"
  ike_version   = 2
}

resource "google_compute_router" "router" {
  name    = "vpn-router"
  network = google_compute_network.vpc.id
  region  = "us-central1"
  bgp {
    asn = 64514
  }
}

resource "google_compute_router_interface" "if0" {
  name       = "interface-0"
  router     = google_compute_router.router.name
  region     = "us-central1"
  ip_range   = "169.254.0.1/30"
  vpn_tunnel = google_compute_vpn_tunnel.tunnel0.name
}

resource "google_compute_router_peer" "peer0" {
  name                      = "peer-0"
  router                    = google_compute_router.router.name
  region                    = "us-central1"
  peer_ip_address           = "169.254.0.2"
  peer_asn                  = 65001
  advertised_route_priority = 100
  interface                 = google_compute_router_interface.if0.name
}

Dedicated Interconnect: When You Need Raw Speed

Dedicated Interconnect gives you direct physical connections from your on-premises router to a GCP colocation facility. You get 10 Gbps or 100 Gbps per link, with an SLA of 99.99% uptime when you have two redundant links. This is for workloads that can't tolerate internet variability: real-time data pipelines, high-frequency trading, or large-scale database replication. The catch: you need to be physically near a GCP colocation facility (over 100+ locations worldwide), and you must order cross-connects from a carrier. Provisioning takes weeks, not hours. You also need to manage BGP sessions and VLAN attachments. Each interconnect supports up to 8 VLAN attachments (each with its own BGP session), which map to different VPCs or regions. For production, always order two links from different carriers or diverse paths to avoid single points of failure.

resource "google_compute_interconnect" "dedicated_interconnect" {
  name                    = "dc-interconnect"
  location                = "https://www.googleapis.com/compute/v1/projects/my-project/global/interconnectLocations/nyc-zone1-1"
  requested_link_count    = 1
  link_type               = "LINK_TYPE_ETHERNET_10G_LR"
  admin_enabled           = true
  description             = "Dedicated Interconnect to NYC DC"
  customer_name           = "My Company"
  interconnect_type       = "DEDICATED"
}

resource "google_compute_interconnect_attachment" "vlan_attachment" {
  name                     = "vlan-attachment-prod"
  interconnect             = google_compute_interconnect.dedicated_interconnect.id
  router                   = google_compute_router.dc_router.id
  mtu                      = 1440
  region                   = "us-east4"
  type                     = "PARTNER"
  edge_availability_domain = "AVAILABILITY_DOMAIN_1"
  candidate_subnets        = ["10.0.0.0/29"]
}

resource "google_compute_router" "dc_router" {
  name    = "dc-router"
  network = google_compute_network.vpc.id
  region  = "us-east4"
  bgp {
    asn = 64514
  }
}

resource "google_compute_router_interface" "vlan_if" {
  name       = "vlan-if"
  router     = google_compute_router.dc_router.name
  region     = "us-east4"
  ip_range   = "169.254.0.1/29"
  vlan_attachment = google_compute_interconnect_attachment.vlan_attachment.name
}

resource "google_compute_router_peer" "vlan_peer" {
  name                      = "vlan-peer"
  router                    = google_compute_router.dc_router.name
  region                    = "us-east4"
  peer_ip_address           = "169.254.0.2"
  peer_asn                  = 65001
  advertised_route_priority = 100
  interface                 = google_compute_router_interface.vlan_if.name
}

Partner Interconnect: The Middle Ground

Partner Interconnect lets you connect to GCP through a supported service provider (like Equinix, Megaport, or CenturyLink). You don't need to be in a GCP colo; the provider handles the last mile. Bandwidth ranges from 50 Mbps to 10 Gbps, and you get an SLA of 99.9% to 99.99% depending on the provider. This is ideal if you're not near a GCP facility or need lower bandwidth than Dedicated Interconnect offers. The tradeoff: you're dependent on the provider's network and support. Provisioning is faster than Dedicated Interconnect (days vs weeks), but you still need to coordinate with the provider. Each VLAN attachment is a Layer 2 connection with a BGP session. For production, use two connections from different providers or diverse paths to avoid provider outages.

resource "google_compute_interconnect_attachment" "partner_attachment" {
  name                     = "partner-attachment-prod"
  router                   = google_compute_router.partner_router.id
  mtu                      = 1440
  region                   = "us-west1"
  type                     = "PARTNER"
  edge_availability_domain = "AVAILABILITY_DOMAIN_1"
  candidate_subnets        = ["10.1.0.0/29"]
  partner_metadata {
    partner_name = "Megaport"
    interconnect_name = "megaport-vlan-123"
  }
}

resource "google_compute_router" "partner_router" {
  name    = "partner-router"
  network = google_compute_network.vpc.id
  region  = "us-west1"
  bgp {
    asn = 64514
  }
}

resource "google_compute_router_interface" "partner_if" {
  name       = "partner-if"
  router     = google_compute_router.partner_router.name
  region     = "us-west1"
  ip_range   = "169.254.1.1/29"
  vlan_attachment = google_compute_interconnect_attachment.partner_attachment.name
}

resource "google_compute_router_peer" "partner_peer" {
  name                      = "partner-peer"
  router                    = google_compute_router.partner_router.name
  region                    = "us-west1"
  peer_ip_address           = "169.254.1.2"
  peer_asn                  = 65001
  advertised_route_priority = 100
  interface                 = google_compute_router_interface.partner_if.name
}

BGP Routing: The Glue That Holds It Together

All three connectivity options use BGP for dynamic routing. GCP's Cloud Router runs BGP on your behalf, exchanging routes with your on-premises router. You must configure BGP sessions on each VLAN attachment or VPN tunnel. Key parameters: ASN (use private ASN 64512-65534 for GCP), peer IPs (link-local addresses from 169.254.0.0/16), and route priority (MED). GCP supports both eBGP and iBGP, but eBGP is standard for hybrid. You can advertise custom IP ranges from your VPC to on-premises, and receive on-premises routes. For production, always use BGP-based routing (not static routes) for automatic failover. Configure route priority to prefer one link over another (e.g., Dedicated Interconnect over VPN). Also, enable graceful restart to minimize disruption during BGP restarts.

# On-premises router (Cisco IOS-XE example)
router bgp 65001
 bgp log-neighbor-changes
 neighbor 169.254.0.1 remote-as 64514
 neighbor 169.254.0.1 description GCP-Cloud-Router
 neighbor 169.254.0.1 ebgp-multihop 2
 neighbor 169.254.0.1 update-source Loopback0
 !
 address-family ipv4
  neighbor 169.254.0.1 activate
  network 10.0.0.0/8
  maximum-paths 2
 exit-address-family
!
# Verify BGP session
show ip bgp summary
show ip route bgp

High Availability and Failover Strategies

Single points of failure are unacceptable in production. For Cloud VPN, use HA VPN with two tunnels in different zones. For Dedicated Interconnect, order two links in different edge availability domains. For Partner Interconnect, use two different providers or diverse paths. GCP's Cloud Router supports active-active and active-passive configurations via BGP route priority (MED). For active-passive, set a higher MED on the backup link. For active-active, use equal MED and enable multipath. Always test failover: simulate a link failure and verify traffic shifts within seconds. Also, consider using VPN as a backup for Interconnect (or vice versa). Monitor BGP session state and alert on flapping. Use Cloud Monitoring to track packet loss and latency on each link.

# Active-passive: Dedicated Interconnect primary, VPN backup
resource "google_compute_router_peer" "primary_peer" {
  name                      = "primary-peer"
  router                    = google_compute_router.main.name
  region                    = "us-east4"
  peer_ip_address           = "169.254.0.2"
  peer_asn                  = 65001
  advertised_route_priority = 100  # Lower priority = preferred
  interface                 = google_compute_router_interface.primary_if.name
}

resource "google_compute_router_peer" "backup_peer" {
  name                      = "backup-peer"
  router                    = google_compute_router.main.name
  region                    = "us-east4"
  peer_ip_address           = "169.254.1.2"
  peer_asn                  = 65002
  advertised_route_priority = 200  # Higher priority = backup
  interface                 = google_compute_router_interface.backup_if.name
}

Performance Tuning: MTU, Latency, and Throughput

Hybrid connections have different performance characteristics. Cloud VPN has an MTU of 1460 bytes due to IPSec overhead. Interconnect attachments default to 1440 bytes (to accommodate VLAN tagging). If your applications use jumbo frames (9000 MTU), you'll see fragmentation. Solution: set the MTU on your on-premises router interfaces to match GCP's, or enable MSS clamping (set TCP MSS to 1436 for VPN, 1400 for Interconnect). Latency on Interconnect is typically 1-5 ms within a region, while VPN adds 5-20 ms depending on distance. Throughput on Dedicated Interconnect is line rate (10/100 Gbps), but VPN is limited to 3 Gbps per tunnel (with HA VPN). For high throughput, use multiple tunnels with ECMP. Monitor throughput with Cloud Monitoring and tools like iperf3.

# Test MTU from GCP VM to on-premises server
# On GCP VM
gcloud compute ssh my-vm --zone=us-central1-a -- 'ping -M do -s 1472 10.0.0.1'
# If successful, MTU >= 1500. For VPN, try 1432 (1460 - 28 ICMP header)
ping -M do -s 1432 10.0.0.1
# Use iperf3 to test throughput
iperf3 -c 10.0.0.1 -t 30 -P 4

Security Considerations: Encryption and Access Control

Cloud VPN encrypts traffic with IPSec (IKEv2, AES-256). Interconnect traffic is not encrypted by default—it's physically isolated but could be snooped if the provider's network is compromised. For sensitive data, use application-level encryption (TLS) or MACsec (if supported). GCP offers MACsec for Dedicated Interconnect at an extra cost. Also, use VPC firewall rules to restrict traffic between on-premises and GCP. Never allow all traffic; use the principle of least privilege. For VPN, rotate preshared keys regularly (every 90 days). For Interconnect, use BGP authentication (MD5) to prevent route hijacking. Monitor for unauthorized BGP announcements with GCP's Route Analyzer.

resource "google_compute_firewall" "allow_on_prem_to_gcp" {
  name    = "allow-on-prem-to-gcp"
  network = google_compute_network.vpc.name
  allow {
    protocol = "tcp"
    ports    = ["443", "3306"]
  }
  source_ranges = ["10.0.0.0/8"]  # On-premises CIDR
  target_tags   = ["web", "db"]
}

resource "google_compute_firewall" "deny_all_other_on_prem" {
  name    = "deny-other-on-prem"
  network = google_compute_network.vpc.name
  deny {
    protocol = "all"
  }
  source_ranges = ["10.0.0.0/8"]
  priority      = 1000
}

Monitoring and Troubleshooting

You can't fix what you can't see. Use Cloud Monitoring to track BGP session state, tunnel status, packet loss, and latency. Set up alerts for BGP session down, high latency (>10ms for Interconnect, >50ms for VPN), and packet loss >0.1%. For VPN, monitor IKE phase status. For Interconnect, monitor optical signal levels if available. Use gcloud commands to check status: gcloud compute vpn-tunnels list, gcloud compute interconnects get-status. For deep troubleshooting, enable VPC Flow Logs to see traffic patterns. Common issues: BGP session flapping (check timers), asymmetric routing (ensure route priorities are consistent), and MTU mismatches (test with ping).

# Check VPN tunnel status
gcloud compute vpn-tunnels describe tunnel-0 --region us-central1 --format="json(status,detailedStatus)"
# Check interconnect status
gcloud compute interconnects get-status dc-interconnect
# Check BGP session
gcloud compute routers get-status vpn-router --region us-central1 --format="json(bgpPeerStatus)"
# Enable VPC Flow Logs on subnet
gcloud compute networks subnets update my-subnet --region us-central1 --enable-flow-logs --logging-aggregation-interval=interval-5-sec --logging-flow-sampling=0.5

Cost Analysis: VPN vs Interconnect

Cloud VPN is cheap: no recurring charges beyond the VM (if using VPN gateway) or per-tunnel fees (HA VPN is free). You pay for egress traffic at standard internet rates (~$0.12/GB). Dedicated Interconnect has a monthly fee per link (~$2,000 for 10G, ~$20,000 for 100G) plus egress at lower rates (~$0.04/GB). Partner Interconnect charges vary by provider but typically $500-$2,000/month per 1G connection. The break-even point: if you transfer more than 20 TB/month, Interconnect is cheaper. But cost isn't everything—consider latency and reliability. For mission-critical workloads, the cost of downtime far exceeds the interconnect fee. Use GCP's pricing calculator to estimate costs based on your traffic patterns.

# Estimate monthly cost for 50 TB egress
# VPN: 50,000 GB * $0.12 = $6,000
# Dedicated Interconnect 10G: $2,000 + 50,000 * $0.04 = $4,000
# Partner Interconnect 1G: $1,000 + 50,000 * $0.06 = $4,000
# Break-even: 20 TB (VPN $2,400 vs Interconnect $2,800 at 10G)
echo "VPN: $6,000"
echo "Dedicated: $4,000"
echo "Partner: $4,000"

Migration Strategy: From VPN to Interconnect

Migrating from VPN to Interconnect requires careful planning to avoid downtime. Steps: 1) Order and provision the interconnect (weeks lead time). 2) Configure BGP on the interconnect with a higher route priority (MED) than the VPN. 3) Verify the interconnect BGP session is up and routes are exchanged. 4) Gradually lower the MED on the interconnect to shift traffic. 5) Monitor for any issues (latency, packet loss). 6) Once traffic is stable, remove the VPN routes or keep VPN as backup. Never cut over all at once—use a phased approach. For active-active, keep both and use ECMP. Document the cutover plan and have a rollback procedure (increase MED on interconnect to revert to VPN).

# Step 4: Shift traffic by lowering MED on interconnect peer
resource "google_compute_router_peer" "interconnect_peer" {
  name                      = "interconnect-peer"
  router                    = google_compute_router.main.name
  region                    = "us-east4"
  peer_ip_address           = "169.254.0.2"
  peer_asn                  = 65001
  advertised_route_priority = 50  # Lower than VPN's 100
  interface                 = google_compute_router_interface.interconnect_if.name
}

# Keep VPN peer with higher priority for backup
resource "google_compute_router_peer" "vpn_peer" {
  name                      = "vpn-peer"
  router                    = google_compute_router.main.name
  region                    = "us-east4"
  peer_ip_address           = "169.254.1.2"
  peer_asn                  = 65002
  advertised_route_priority = 200
  interface                 = google_compute_router_interface.vpn_if.name
}

Advanced: Multi-Cloud and Global Hybrid Connectivity

GCP hybrid connectivity can extend to multi-cloud setups. You can connect on-premises to GCP, then use GCP's Cross-Cloud Interconnect to reach AWS or Azure. Or, use a VPN between clouds. For global hybrid, use GCP's Cloud Router with BGP to advertise routes across regions. Consider using a transit VPC with a centralized Cloud Router to manage multiple interconnects. For latency-sensitive global applications, use GCP's Premium Tier networking to route traffic over Google's backbone. Also, consider using Network Connectivity Center (NCC) to manage multiple hybrid connections from a single hub. NCC supports spoke VPCs and on-premises networks, simplifying routing and policy management.

# Cross-Cloud Interconnect to AWS (requires partner)
resource "google_compute_interconnect_attachment" "aws_attachment" {
  name                     = "aws-attachment"
  router                   = google_compute_router.global_router.id
  mtu                      = 1440
  region                   = "us-west1"
  type                     = "PARTNER"
  edge_availability_domain = "AVAILABILITY_DOMAIN_1"
  candidate_subnets        = ["10.2.0.0/29"]
  partner_metadata {
    partner_name = "Megaport"
    interconnect_name = "megaport-aws-vlan"
  }
}

# Network Connectivity Center hub
resource "google_network_connectivity_hub" "hub" {
  name        = "global-hub"
  description = "Hub for all hybrid connections"
}

resource "google_network_connectivity_spoke" "on_prem_spoke" {
  name        = "on-prem-spoke"
  hub         = google_network_connectivity_hub.hub.id
  linked_vpn_tunnels {
    uris = [google_compute_vpn_tunnel.tunnel0.self_link]
  }
}

Network Connectivity Center: Centralized Hybrid Management

As your hybrid footprint grows, managing individual VPN tunnels and VLAN attachments across multiple VPCs becomes unwieldy. Network Connectivity Center (NCC) is an orchestration framework that provides a central hub for managing connectivity between VPC spokes, on-premises networks via VPN or Interconnect, and router appliances. Instead of configuring BGP sessions for each connection pair, you add spokes to an NCC hub and let it handle route exchange. NCC supports VPC spokes (for inter-VPC routing) and hybrid spokes (for VPN tunnels and VLAN attachments). It automatically propagates routes between spokes, reducing configuration complexity and eliminating the need for transitive VPC peering. Key benefits: simplified routing, centralized policy management, and support for up to 2000 spokes per hub. However, NCC adds a small amount of latency and has specific routing limits. For large-scale deployments, we've found NCC reduces BGP configuration overhead by 80%.

resource "google_network_connectivity_hub" "hybrid_hub" {
  name        = "global-hybrid-hub"
  description = "Central hub for all hybrid connections"
}

resource "google_network_connectivity_spoke" "vpc_spoke" {
  name        = "prod-vpc-spoke"
  hub         = google_network_connectivity_hub.hybrid_hub.id
  linked_vpc_network {
    uri       = google_compute_network.prod_vpc.self_link
    exclude_export_ranges = ["0.0.0.0/0"]
  }
}

resource "google_network_connectivity_spoke" "vpn_spoke" {
  name        = "on-prem-vpn-spoke"
  hub         = google_network_connectivity_hub.hybrid_hub.id
  linked_vpn_tunnels {
    uris      = [google_compute_vpn_tunnel.tunnel0.self_link]
  }
}

HA VPN Over Cloud Interconnect: Encryption on Dedicated Links

Dedicated Interconnect traffic is not encrypted by default. For workloads requiring encryption at the network layer (e.g., PCI DSS, HIPAA), you can deploy HA VPN on top of your Cloud Interconnect VLAN attachments. This combines the low latency and high bandwidth of Interconnect with IPSec encryption. However, there's a throughput tradeoff: each HA VPN tunnel maxes out at 3 Gbps, so matching a 10 Gbps VLAN attachment requires 4 tunnels. For a 100 Gbps link, you'd need 34 tunnels. GCP's HA VPN over Interconnect wizard automates gateway and tunnel creation based on your VLAN capacity. BGP sessions run over each tunnel, and you can use ECMP across multiple tunnels for higher throughput. Key considerations: encryption adds CPU overhead on your peer router, and you need to manage pre-shared keys (or use IKEv2 certificates). In production, we use HA VPN over Interconnect for regulated workloads and plain Interconnect for everything else. The 3 Gbps per-tunnel limit is a real constraint — plan tunnel count carefully.

resource "google_compute_ha_vpn_gateway" "encrypted_gateway" {
  name    = "encrypted-gw"
  network = google_compute_network.vpc.id
  region  = "us-central1"
  vpn_interconnect_attachments = [
    google_compute_interconnect_attachment.vlan_attachment.id
  ]
}

resource "google_compute_vpn_tunnel" "tunnel_0" {
  name          = "encrypted-tunnel-0"
  region        = "us-central1"
  vpn_gateway   = google_compute_ha_vpn_gateway.encrypted_gateway.id
  peer_external_gateway = google_compute_external_vpn_gateway.on_prem.id
  interface     = 0
  ike_version   = 2
  shared_secret = random_password.psk.result
  vpn_gateway_interface = 0
}