Google Cloud — Memorystore (Redis & Memcached)

Memorystore provides fully managed in-memory data stores for Redis and Memcached on Google Cloud. It eliminates the operational burden of self-managing caching clusters while offering high availability, automatic failover, and integrated security. Choosing between Redis and Memcached depends on your data structure needs, persistence requirements, and throughput demands.

Why Memorystore Exists: The Cache Layer Problem

In production, your database is the bottleneck. Every query that hits PostgreSQL or MySQL adds latency and contention. The solution is a cache layer — but running your own Redis or Memcached cluster is a nightmare. You need to handle failover, patching, monitoring, and scaling. Google Cloud Memorystore removes that operational burden. It provides managed Redis and Memcached instances with automatic replication, failover, and maintenance. The trade-off? You lose some control over configuration and pay a premium. For most teams, that's a win. Memorystore integrates natively with Compute Engine, GKE, and Cloud Run via the VPC, keeping latency sub-millisecond. It's not just a cache; it's a session store, rate limiter backend, and pub/sub broker. But don't treat it as a database — it's ephemeral by design. Data persists only if you enable persistence (AOF or RDB), and even then, it's not a replacement for durable storage. Use it for hot data that can be regenerated.

gcloud redis instances create my-cache \
  --size=5 \
  --region=us-central1 \
  --redis-version=redis_7_0 \
  --tier=standard \
  --connect-mode=direct-peering \
  --network=default

Choosing Between Redis and Memcached

Redis and Memcached are both in-memory data stores, but they serve different purposes. Memcached is a simple key-value cache with no persistence, no replication, and no data structures beyond strings. It's fast and simple — ideal for caching database query results or API responses where data loss is acceptable. Redis, on the other hand, is a feature-rich data structure server. It supports lists, sets, sorted sets, hashes, streams, and pub/sub. It can persist data to disk, replicate across nodes, and run Lua scripts. For most production workloads, Redis is the better choice. Use Memcached only if you need pure caching with minimal overhead and can tolerate data loss. Memorystore offers both, but Redis has more configuration options (persistence, replication, maintenance windows). Memcached is simpler but less flexible. If you're unsure, start with Redis. The cost difference is negligible for small instances, and Redis's features will save you from building custom solutions later.

import redis
import bmemcached

# Redis client
r = redis.Redis(host='10.0.0.3', port=6379, decode_responses=True)
r.set('key', 'value')
print(r.get('key'))  # 'value'

# Memcached client
mc = bmemcached.Client(['10.0.0.4:11211'])
mc.set('key', 'value')
print(mc.get('key'))  # 'value'

Network Architecture: VPC Peering and Private IP

Memorystore instances are not publicly accessible. They live inside a Google-managed VPC and connect to your resources via VPC peering or direct peering. When you create an instance, you specify a network (default or custom). Memorystore then provisions a private IP address in that network's subnet. Your Compute Engine VMs, GKE pods, and Cloud Run services can reach it if they are in the same VPC or use Serverless VPC Access. The latency is typically under 1ms within the same region. For cross-region access, you need VPC peering or Cloud VPN, but latency increases. Never expose Memorystore to the internet — use a proxy or Cloud NAT if absolutely necessary. Also, note that Memorystore uses a /29 subnet from your VPC's IP range. Plan your IP space accordingly to avoid conflicts. If you use Shared VPC, ensure the host project has sufficient IP space.

gcloud redis instances describe my-cache --region=us-central1 \
  --format='value(authorizedNetwork)'
# Output: projects/my-project/global/networks/default

# Verify connectivity from a VM
gcloud compute ssh my-vm --zone=us-central1-a -- 'redis-cli -h 10.0.0.3 ping'

Configuring Persistence: AOF vs RDB

By default, Memorystore Redis instances are ephemeral — data is lost on failover or restart. To persist data, enable AOF (Append-Only File) or RDB (Redis Database Backup). AOF logs every write operation, providing near-real-time durability. RDB takes periodic snapshots. AOF is safer but uses more memory and disk I/O. RDB is lighter but can lose data between snapshots. In Memorystore, you can enable persistence during creation or update. For production, use AOF with fsync every second. This gives you at most 1 second of data loss. RDB is acceptable for caching where data can be regenerated. Note that persistence adds latency — test your workload. Also, Memorystore stores persistence data in Google Cloud Storage, not on the instance. This means failover is fast because the new primary reads from GCS. But it also means you pay for storage egress if you export backups.

gcloud redis instances update my-cache \
  --region=us-central1 \
  --persistence-mode=aof \
  --aof-persistence-config='append-fsync=everysec'

Scaling Memorystore: Vertical and Horizontal

Memorystore supports vertical scaling (changing instance size) and horizontal scaling (adding replicas). Vertical scaling is straightforward — update the size parameter. The instance restarts with a brief downtime (seconds to minutes depending on size). For zero-downtime scaling, use a blue-green deployment: create a new instance, migrate data, switch traffic. Horizontal scaling with replicas improves read throughput and provides failover. In Standard tier, you get one primary and one replica. You can add more replicas (up to 5) for read-heavy workloads. Writes still go to the primary. For Redis, you can also use sharding with Cluster mode (Memorystore for Redis Cluster). This partitions data across multiple shards, each with a primary and replicas. Cluster mode is complex but necessary for datasets larger than 300GB or write throughput beyond a single instance. Plan carefully — cluster mode requires application changes (Redis Cluster client).

# Vertical scale: increase size to 10GB
gcloud redis instances update my-cache \
  --region=us-central1 \
  --size=10

# Add replicas (Standard tier only)
gcloud redis instances update my-cache \
  --region=us-central1 \
  --replica-count=3

Monitoring and Alerting: The Metrics That Matter

Memorystore exposes metrics via Cloud Monitoring. Key metrics: CPU utilization, memory usage, keyspace hits/misses, connected clients, replication lag, and evictions. Set alerts on CPU > 80%, memory > 80%, evictions > 0, and replication lag > 10 seconds. Evictions mean your instance is too small — increase size or optimize TTLs. High replication lag indicates network issues or overloaded primary. Also monitor redis_connected_clients — if it spikes, you may have connection leaks. Use the redis-cli INFO command for deeper diagnostics. For Memcached, monitor curr_items, evictions, and get_hits. Set up a dashboard with these metrics. Don't forget to monitor the instance's health — Memorystore automatically fails over if the primary becomes unhealthy, but you should know when it happens.

# Get Redis INFO from a VM
redis-cli -h 10.0.0.3 INFO stats | grep -E 'keyspace_hits|keyspace_misses|evicted_keys'

# Create a Cloud Monitoring alert via gcloud
gcloud alpha monitoring policies create \
  --policy-from-file=alert-policy.yaml

Backup and Restore: Exporting and Importing Data

Memorystore allows you to export Redis data to a GCS bucket (RDB format) and import from GCS. This is useful for backups, migration, or seeding a new instance. Exports are manual or scheduled via Cloud Scheduler + Cloud Functions. There's no built-in scheduled backup — you must build it. The export process creates an RDB file in the bucket. Importing loads that file into a new or existing instance. Note: import overwrites all existing data. For incremental backups, you need to use AOF persistence and export the AOF file, but Memorystore doesn't support AOF export directly. Instead, use Redis replication to a secondary instance and export from there. Also, exports consume CPU and memory — do them during low traffic. Test your restore process regularly. We've seen teams assume backups work only to find corrupted files during a disaster.

# Export to GCS
gcloud redis instances export my-cache \
  --region=us-central1 \
  --output-path=gs://my-bucket/backups/my-cache.rdb

# Import from GCS
gcloud redis instances import my-cache \
  --region=us-central1 \
  --input-path=gs://my-bucket/backups/my-cache.rdb

Security: IAM, VPC, and Encryption

Memorystore security starts with IAM. Use roles like redis.editor and redis.viewer to control access to instance management. For data access, Memorystore supports Redis AUTH (password) — always enable it. The password is set at instance creation and can be rotated. Also, use VPC firewall rules to restrict traffic to only your application VMs. Memorystore does not support customer-managed encryption keys (CMEK) for data at rest — it uses Google-managed encryption. For data in transit, TLS is not supported natively for Redis; you must use a proxy like Envoy or HAProxy to add TLS. Memcached doesn't support TLS either. If you need encryption in transit, consider using Cloud VPN or VPC peering with private IPs, which are not exposed to the internet. Also, enable VPC Service Controls to prevent data exfiltration.

# Create instance with AUTH enabled
gcloud redis instances create my-cache \
  --region=us-central1 \
  --redis-config='requirepass=my-secure-password'

# Connect with AUTH
redis-cli -h 10.0.0.3 -a 'my-secure-password' ping

High Availability and Disaster Recovery

Memorystore Standard tier provides automatic failover within the same region across zones. If the primary fails, a replica is promoted. This takes seconds. For disaster recovery across regions, you need to set up cross-region replication manually. Memorystore does not support native cross-region replication. Use Redis replication with a secondary instance in another region, or use a multi-region deployment with application-level routing. For Memcached, there is no replication — data loss is expected. For Redis, you can configure a replica in another region using the gcloud redis instances create with --replica-of flag. This creates an async replication link. The replica is read-only. If the primary region goes down, you can promote the replica by detaching it. This is a manual process. Also, consider using Memorystore for Redis Cluster with multi-region sharding, but that's complex. Simpler: use a global cache with a single primary and read replicas in other regions.

# Create a replica in another region
gcloud redis instances create my-cache-replica \
  --region=europe-west1 \
  --replica-of=projects/my-project/locations/us-central1/instances/my-cache \
  --tier=standard

# Promote replica to primary (detach)
gcloud redis instances update my-cache-replica \
  --region=europe-west1 \
  --clear-replica-of

Cost Optimization: Rightsizing and Reserved Instances

Memorystore pricing is based on instance size, tier, and region. Standard tier costs roughly 2x Basic tier. To optimize costs, start with the smallest instance that meets your performance needs. Monitor memory usage and evictions — if memory usage is below 50%, consider downsizing. Use reserved instances (1 or 3 year commitments) for predictable workloads to save up to 40%. For bursty workloads, use Basic tier with auto-scaling (not natively supported — you need to script it). Also, consider using a single larger Redis instance with multiple databases (select DB index) instead of many small instances. This reduces overhead and cost. For Memcached, you can use a single large instance as well. Avoid over-provisioning — Memorystore instances are billed per hour, so idle instances waste money. Use labels to track costs per team or environment.

# List instances with size and tier
gcloud redis instances list --region=us-central1 \
  --format='table(name, sizeGb, tier, redisVersion)'

# Estimate monthly cost (example)
echo "Instance: my-cache, Size: 5GB, Tier: Standard, Cost: ~$150/month"

Troubleshooting Common Issues

Common Memorystore issues include connection timeouts, high latency, and out-of-memory errors. Connection timeouts often stem from VPC firewall rules blocking port 6379/11211. Check firewall rules and ensure the client's network has VPC peering. High latency can be due to network congestion, large payloads, or slow commands (e.g., KEYS, SMEMBERS). Use the --latency flag in redis-cli to measure. Out-of-memory errors trigger evictions or OOM kills. Monitor used_memory and set maxmemory-policy to allkeys-lru to avoid crashes. Another issue: replication lag. Check master_repl_offset and slave_repl_offset. If lag grows, the replica may be overloaded or network is slow. For Memcached, common issues are connection limits (default 1024) and slab calcification. Increase -c for connections and adjust slab sizes. Always check logs in Cloud Logging for error messages.

# Measure latency from a VM
redis-cli -h 10.0.0.3 --latency -a 'password'

# Check replication lag
redis-cli -h 10.0.0.3 INFO replication | grep -E 'master_repl_offset|slave_repl_offset'

Migration Strategies: Moving to Memorystore

Migrating an existing Redis or Memcached workload to Memorystore requires planning. For Redis, the simplest method is to set up replication from your existing instance to Memorystore. Use the SLAVEOF command or configure Memorystore as a replica of your existing instance (if it's accessible). Once replication is caught up, promote Memorystore to primary. For offline migration, export your data as RDB and import into Memorystore. This requires downtime. For Memcached, there's no replication — you must drain the cache and repopulate. Use a dual-write strategy: write to both old and new caches, then switch reads. For large datasets, use a blue-green deployment: create a new Memorystore instance, migrate traffic gradually. Always test with a subset of traffic first. Rollback plan: keep the old instance running for a few days.

# On existing Redis, set up replication to Memorystore
redis-cli -h <existing-redis> SLAVEOF 10.0.0.3 6379

# Check replication status
redis-cli -h <existing-redis> INFO replication

# When ready, promote Memorystore by stopping replication
redis-cli -h <existing-redis> SLAVEOF NO ONE