Google Cloud — Dataproc (Hadoop & Spark)

Dataproc is a managed Spark and Hadoop service that enables fast, cost-effective cluster creation and management. It integrates natively with Cloud Storage (via the gs:// connector), BigQuery, and Cloud Bigtable. Autoscaling and ephemeral clusters help control costs by shutting down idle resources automatically.

Why Dataproc Exists: The Hadoop/Spark Managed Service

Google Cloud Dataproc is a managed service for running Apache Hadoop and Apache Spark clusters. It abstracts away the operational overhead of provisioning, configuring, and tuning clusters, letting you focus on data processing pipelines. Dataproc integrates natively with Cloud Storage (instead of HDFS), BigQuery, and other GCP services. It's designed for ephemeral clusters — spin up, run jobs, tear down — which reduces cost and complexity. For production workloads, Dataproc offers features like cluster autoscaling, preemptible instances, and component gateway for secure access to UIs. The key differentiator from self-managed clusters is the automation of lifecycle management, including automatic image updates and node recovery. However, this abstraction comes with trade-offs: you lose fine-grained control over YARN configurations and OS-level tuning. Understanding when to use Dataproc vs. Compute Engine-based clusters is critical for cost and performance optimization.

gcloud dataproc clusters create my-cluster \
  --region=us-central1 \
  --zone=us-central1-a \
  --master-machine-type=n1-standard-4 \
  --worker-machine-type=n1-standard-4 \
  --num-workers=2 \
  --image-version=2.1-debian11 \
  --optional-components=JUPYTER \
  --enable-component-gateway \
  --bucket=my-dataproc-staging-bucket \
  --scopes=cloud-platform

Cluster Architecture: Master, Workers, and Preemptibles

A Dataproc cluster consists of one master node (or three for HA) and multiple worker nodes. The master runs YARN ResourceManager, HDFS NameNode (if using HDFS), and Spark driver. Workers run YARN NodeManager and HDFS DataNode. For cost optimization, you can mix standard and preemptible (preemptible) workers. Preemptible instances are up to 80% cheaper but can be terminated at any time. Use them for fault-tolerant, idempotent tasks like map-only jobs or Spark stages with checkpointing. Never use preemptibles for driver nodes or stateful workloads. Dataproc also supports secondary worker groups for elastic scaling. The cluster's network configuration matters: use VPC-native clusters with private IPs and Cloud NAT for internet access. For security, enable Kerberos or use IAM-based access control. The component gateway provides HTTPS access to UIs like YARN, Spark History Server, and Jupyter.

gcloud dataproc clusters create my-cluster \
  --region=us-central1 \
  --master-machine-type=n1-standard-4 \
  --worker-machine-type=n1-standard-4 \
  --num-workers=2 \
  --num-preemptible-workers=10 \
  --preemptible-worker-boot-disk-size=100GB \
  --image-version=2.1-debian11 \
  --enable-component-gateway

Initialization Actions: Customizing Your Cluster

Initialization actions are scripts that run on every node during cluster creation. They allow you to install custom software, configure OS settings, or mount persistent disks. Common use cases: installing Python packages, configuring Kerberos, or setting up monitoring agents. Scripts must be idempotent and handle failures gracefully. Store them in Cloud Storage and reference them in the create command. You can pass arguments via metadata keys. For production, version your init scripts and test them in a non-production environment. Avoid long-running init actions (timeout is 10 minutes). Use them sparingly — prefer Dataproc optional components or custom images for complex setups. Example: install the BigQuery connector and configure Spark to use it.

#!/bin/bash
set -euxo pipefail

ROLE=$(/usr/share/google/get_metadata_value attributes/dataproc-role)
if [[ "${ROLE}" == 'Master' ]]; then
  # Install BigQuery connector
  wget -q https://storage.googleapis.com/spark-lib/bigquery/spark-bigquery-latest_2.12.jar \
    -P /usr/lib/spark/jars/
  
  # Configure Spark defaults
  cat >> /etc/spark/conf/spark-defaults.conf <<EOF
spark.hadoop.fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem
spark.hadoop.fs.AbstractFileSystem.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS
EOF
fi

Submitting Jobs: Spark, Hive, and More

Dataproc supports multiple job types: Spark, PySpark, SparkR, Hive, Hadoop, Pig, and Presto. Jobs are submitted via gcloud, REST API, or the Console. For production, use the gcloud command or API in CI/CD pipelines. Jobs run on the cluster and output logs to Cloud Logging. You can specify job parameters, JAR files, and properties. For Spark, use the --properties flag to set Spark configurations like executor memory. Always set spark.dynamicAllocation.enabled=true for better resource utilization. For Hive, use the Metastore service for persistent metadata. Jobs can be submitted to running clusters or created with a workflow template. Workflow templates allow you to define a DAG of jobs with dependencies and retries. This is the preferred way for production pipelines.

gcloud dataproc jobs submit spark \
  --cluster=my-cluster \
  --region=us-central1 \
  --class=com.example.MySparkApp \
  --jars=gs://my-bucket/jars/myapp.jar \
  --properties=spark.executor.instances=4,spark.executor.memory=4g,spark.dynamicAllocation.enabled=true \
  -- gs://input/data.csv gs://output/results

Autoscaling: Dynamic Worker Management

Dataproc autoscaling adjusts the number of worker nodes based on YARN memory and CPU utilization. It uses a policy you define with min/max instances, scale-up/down factors, and cooldown periods. Autoscaling works best for variable workloads. It does not support preemptible instances in the primary group; use secondary worker groups for preemptibles. For production, set realistic min/max limits to avoid cost spikes. Monitor autoscaling events in Cloud Logging. Common pitfalls: scale-down can kill nodes with running tasks, causing job failures. Use graceful decommissioning (enabled by default) to let tasks finish. Autoscaling is not suitable for latency-sensitive or stateful workloads. For those, use fixed-size clusters.

gcloud dataproc autoscaling-policies create my-policy \
  --region=us-central1 \
  --min-instances=2 \
  --max-instances=20 \
  --scale-up-factor=0.05 \
  --scale-down-factor=0.05 \
  --graceful-decommission-timeout=1h \
  --basic-algorithm=yarn \
  --yarn-config={"scaleUpFactor":0.05,"scaleDownFactor":0.05,"gracefulDecommissionTimeout":"3600s"}

Monitoring and Logging: Cloud Monitoring and Logging

Dataproc integrates with Cloud Monitoring and Cloud Logging. Key metrics: YARN memory, CPU utilization, HDFS usage, and job durations. Create dashboards for cluster health and job performance. Set up alerts for high memory pressure, job failures, and cluster idle time. Use Cloud Logging to view job logs and init script logs. For Spark, enable the Spark History Server via component gateway to view event logs. Store Spark event logs in Cloud Storage for long-term retention. Use the Dataproc Agent logs for node-level issues. For production, export logs to BigQuery for analysis. Monitor preemptible instance preemption rates; high rates indicate need for more standard workers.

# Create a custom dashboard using gcloud (simplified)
gcloud monitoring dashboards create \
  --dashboard-json='{
    "displayName": "Dataproc Cluster Dashboard",
    "gridLayout": {
      "widgets": [
        {
          "title": "YARN Memory",
          "xyChart": {
            "dataSets": [{
              "timeSeriesQuery": {
                "timeSeriesFilter": {
                  "filter": "metric.type=\"dataproc.googleapis.com/cluster/yarn/memory\""
                }
              }
            }]
          }
        }
      ]
    }
  }'

Security: IAM, Kerberos, and VPC-SC

Dataproc security involves multiple layers: IAM for API access, Kerberos for cluster internal authentication, and VPC Service Controls for data exfiltration prevention. IAM roles like dataproc.editor allow cluster management. For job submission, use dataproc.jobRunner. Kerberos enables mutual authentication between nodes and services. Enable it for multi-tenant clusters. VPC-SC prevents data from leaving a defined perimeter. For production, use private clusters (no public IPs) and Cloud NAT for outbound access. Encrypt disks with CMEK. Use OS Login for SSH access. Audit all actions with Cloud Audit Logs. Never use default service accounts with broad scopes; create custom service accounts with minimal permissions.

gcloud dataproc clusters create secure-cluster \
  --region=us-central1 \
  --master-machine-type=n1-standard-4 \
  --worker-machine-type=n1-standard-4 \
  --num-workers=2 \
  --image-version=2.1-debian11 \
  --no-address \
  --service-account=my-sa@project.iam.gserviceaccount.com \
  --scopes=cloud-platform \
  --kerberos-config-file=gs://my-bucket/kerberos-config.yaml

Workflow Templates: Orchestrating Pipelines

Workflow templates define a directed acyclic graph (DAG) of Dataproc jobs. They support job dependencies, retries, and timeouts. Use them for production pipelines instead of manual job submission. Templates can be instantiated on-demand or scheduled via Cloud Scheduler. Each job in the template can have different parameters. You can also use workflow templates to create clusters on-the-fly (ephemeral clusters). This is the recommended pattern: create cluster, run jobs, delete cluster. For complex pipelines, consider using Cloud Composer (Airflow) for more flexibility. Workflow templates are idempotent; you can retry the entire workflow on failure.

gcloud dataproc workflow-templates create my-workflow \
  --region=us-central1

gcloud dataproc workflow-templates add-job spark \
  --step-id=step1 \
  --workflow-template=my-workflow \
  --region=us-central1 \
  --class=com.example.ETLJob \
  --jars=gs://my-bucket/jars/etl.jar

gcloud dataproc workflow-templates add-job spark \
  --step-id=step2 \
  --workflow-template=my-workflow \
  --region=us-central1 \
  --class=com.example.AnalyticsJob \
  --jars=gs://my-bucket/jars/analytics.jar \
  --dependencies=step1

gcloud dataproc workflow-templates instantiate my-workflow \
  --region=us-central1

Cost Optimization: Preemptibles, Autoscaling, and Ephemeral Clusters

Dataproc cost is driven by compute (VM instances), storage (persistent disks), and licensing (premium images). To optimize: use preemptible instances for fault-tolerant workloads, enable autoscaling for variable loads, and always use ephemeral clusters. Delete clusters when not in use. Use committed use discounts for baseline capacity. Choose machine types wisely: n1-standard for balanced workloads, n2-highmem for memory-intensive Spark. Use local SSDs for shuffle data (ephemeral). Monitor cost with Billing reports and set budget alerts. Avoid leaving clusters idle; use cluster scheduling to auto-delete. For storage, use Cloud Storage instead of HDFS to avoid persistent disk costs.

gcloud dataproc clusters delete my-cluster \
  --region=us-central1 \
  --quiet

Troubleshooting Common Failures

Common Dataproc failures include: out-of-memory errors, disk full, preemptible node loss, and misconfigured init scripts. For OOM, increase executor memory or reduce partitions. Use spark.memory.fraction and spark.memory.storageFraction. For disk full, use Cloud Storage for data and local SSDs for shuffle. For preemptible loss, enable speculative execution and checkpointing. For init script failures, check /var/log/dataproc-initialization-script-*.log. For job failures, check Cloud Logging and Spark History Server. Use the Dataproc Diagnose tool to collect logs. Common network issues: VPC firewall rules blocking internal communication. Ensure all nodes can communicate on ports 8088, 9870, etc.

gcloud dataproc clusters diagnose my-cluster \
  --region=us-central1 \
  --output-dir=gs://my-bucket/diagnose/

Migrating from On-Premise Hadoop/Spark to Dataproc

Migrating to Dataproc involves moving data, code, and configurations. Data: transfer from HDFS to Cloud Storage using DistCp or Storage Transfer Service. Code: adapt Spark/Hive jobs to use GCS connectors instead of HDFS. Configurations: port YARN and Spark settings to Dataproc properties. Use the same image version to minimize compatibility issues. Test with a small dataset first. Common pitfalls: hardcoded HDFS paths, Kerberos configuration differences, and different default settings. Use the Dataproc Migration Assessment tool to identify issues. For Hive, migrate metadata to Hive Metastore service. For Spark, update dependencies to use GCS connector. Plan for a cutover window and have rollback plan.

hadoop distcp \
  -Dfs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem \
  -Dfs.AbstractFileSystem.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS \
  hdfs://namenode:8020/user/hive/warehouse \
  gs://my-bucket/hive-warehouse/

Advanced: GPU Support and Custom Images

Dataproc supports attaching GPUs to worker nodes for accelerated computing (e.g., Spark ML with RAPIDS). Use accelerator-optimized machine types like n1-standard-8 with 1 GPU. Install NVIDIA drivers via init actions. For custom images, use Dataproc Image Builder to create a base image with pre-installed software. This reduces cluster creation time. Custom images are versioned and stored in Compute Engine. Use them when init actions are too slow or complex. For production, automate image building in CI/CD. Test images in a staging environment. GPU clusters require specific configurations: enable the NVIDIA driver, set Spark to use GPU resources, and configure RAPIDS libraries.

gcloud dataproc clusters create gpu-cluster \
  --region=us-central1 \
  --master-machine-type=n1-standard-8 \
  --worker-machine-type=n1-standard-8 \
  --num-workers=2 \
  --worker-accelerator=type=nvidia-tesla-t4,count=1 \
  --image-version=2.1-debian11 \
  --initialization-actions=gs://goog-dataproc-initialization-actions-us-central1/gpu/install_gpu_driver.sh