Snowpark Container Services: Deploy Jobs, Services & Apps on Snowflake

Imagine you’ve built a machine learning model that predicts customer churn. Traditionally, you’d export data from Snowflake, train the model elsewhere, and then import predictions back. This data shuffle is slow, expensive, and insecure. Snowpark Container Services (SPCS) changes the game: you can run containerized applications—Python, R, Java, or any language—directly inside Snowflake, with no data movement. SPCS supports three workload types: Jobs for batch processing (e.g., daily ETL), Services for long-running endpoints (e.g., REST APIs), and Apps for interactive UIs (e.g., Streamlit dashboards). In this tutorial, you’ll learn how to build, deploy, and debug each type using real-world examples. By the end, you’ll be able to run custom code on Snowflake’s compute, leveraging its security and scalability. Let’s dive in.

Understanding Snowpark Container Services Architecture

Snowpark Container Services (SPCS) is a fully managed container orchestration platform integrated into Snowflake. It allows you to run Docker containers on Snowflake’s compute infrastructure, with native access to Snowflake data and security. The architecture consists of:

• Compute Pool: A set of Snowflake nodes (like a warehouse) where containers run. You define the instance type (e.g., CPU, memory) and scaling parameters.
• Image Repository: A Snowflake-managed registry (or external) where you push your Docker images. You can use Snowflake’s built-in registry or connect to Docker Hub or Amazon ECR.
• Service: A long-running container that exposes an endpoint (e.g., REST API). It can scale horizontally and supports health checks.
• Job: A short-lived container that runs to completion. Ideal for batch processing, ETL, or training models.
• App: An interactive web application (e.g., Streamlit) that users can access via a URL.

All containers run in a secure, isolated environment with network policies and role-based access control (RBAC). Data access is handled via Snowflake’s standard permissions—no need to manage credentials inside the container.

Let’s start by setting up your environment.

-- Create a compute pool (if not exists)
CREATE COMPUTE POOL my_pool
  MIN_NODES = 1
  MAX_NODES = 5
  INSTANCE_FAMILY = CPU_X64_XS;

-- Create an image repository (if not exists)
CREATE IMAGE REPOSITORY my_repo;

-- Grant usage to role
GRANT USAGE ON COMPUTE POOL my_pool TO ROLE my_role;
GRANT USAGE ON IMAGE REPOSITORY my_repo TO ROLE my_role;

Building and Pushing a Docker Image to Snowflake

Before deploying, you need a Docker image. Let’s create a simple Python service that returns the current time. We’ll use Snowflake’s image registry.

Step 1: Create a Dockerfile

Create a file named Dockerfile with the following content:

``dockerfile FROM python:3.9-slim WORKDIR /app COPY requirements.txt . RUN pip install -r requirements.txt COPY app.py . CMD ["python", "app.py"] ``

Step 2: Create app.py

```python from flask import Flask, jsonify import datetime

app = Flask(__name__)

@app.route('/time') def get_time(): return jsonify({'time': datetime.datetime.now().isoformat()})

if __name__ == '__main__': app.run(host='0.0.0.0', port=8080) ```

Step 3: Create requirements.txt

`` flask==2.3.2 ``

Step 4: Build and tag the image

``bash docker build -t my_repo/my_service:latest . ``

Step 5: Push to Snowflake

First, authenticate Docker to Snowflake’s registry. Get the registry URL from Snowflake:

``sql SHOW IMAGE REPOSITORIES; ``

Then, log in using SnowSQL or your Snowflake account credentials:

``bash docker login <registry_url> -u <username> --password-stdin ``

Push the image:

``bash docker push <registry_url>/my_repo/my_service:latest ``

Now your image is stored in Snowflake and ready to deploy.

# Build image
docker build -t my_service:latest .

# Tag for Snowflake registry (replace with your registry URL)
docker tag my_service:latest <registry_url>/my_repo/my_service:latest

# Login (use Snowflake username/password or key pair)
docker login <registry_url> -u <username> --password-stdin

# Push
docker push <registry_url>/my_repo/my_service:latest

Deploying a Long-Running Service

Now let’s deploy the time service as a long-running service. You’ll need a service specification file (YAML).

Step 1: Create service specification

Create service_spec.yaml:

``yaml spec: containers: - name: main image: /my_repo/my_service:latest env: SERVER_PORT: 8080 readinessProbe: port: 8080 path: /time endpoints: - name: time-endpoint port: 8080 public: true ``

Step 2: Create the service

Use SQL to create the service:

``sql CREATE SERVICE my_time_service IN COMPUTE POOL my_pool FROM SPECIFICATION $$ spec: containers: - name: main image: /my_repo/my_service:latest env: SERVER_PORT: 8080 readinessProbe: port: 8080 path: /time endpoints: - name: time-endpoint port: 8080 public: true $$ MIN_INSTANCES = 1 MAX_INSTANCES = 3; ``

Step 3: Check service status

``sql CALL SYSTEM$GET_SERVICE_STATUS('my_time_service'); ``

Step 4: Get the endpoint URL

``sql SHOW ENDPOINTS IN SERVICE my_time_service; ``

Copy the ingress URL and test it in your browser or curl:

``bash curl https://<ingress_url>/time ``

You should see a JSON response with the current time.

Step 5: Update the service

To update the image or spec, use ALTER SERVICE:

``sql ALTER SERVICE my_time_service FROM SPECIFICATION $$ ... updated spec ... $$; ``

Snowflake performs a rolling update.

-- Create service
CREATE SERVICE my_time_service
  IN COMPUTE POOL my_pool
  FROM SPECIFICATION $$
  spec:
    containers:
      - name: main
        image: /my_repo/my_service:latest
        env:
          SERVER_PORT: 8080
        readinessProbe:
          port: 8080
          path: /time
    endpoints:
      - name: time-endpoint
        port: 8080
        public: true
  $$
  MIN_INSTANCES = 1
  MAX_INSTANCES = 3;

-- Check status
CALL SYSTEM$GET_SERVICE_STATUS('my_time_service');

-- Show endpoints
SHOW ENDPOINTS IN SERVICE my_time_service;

Running Batch Jobs with Snowpark Container Services

Jobs are ideal for batch processing, such as daily data transformations or model retraining. Unlike services, jobs run to completion and then stop.

Example: A job that reads data from a Snowflake table and writes results

Create a Python script job.py that uses Snowpark to query data:

```python from snowflake.snowpark import Session import sys

# Get arguments from environment input_table = sys.argv[1] output_table = sys.argv[2]

session = Session.builder.configs({ "connection_name": "snowflake_default" }).create()

df = session.table(input_table) result = df.group_by("category").count() result.write.mode("overwrite").save_as_table(output_table)

session.close() ```

Dockerfile for job:

``dockerfile FROM python:3.9-slim WORKDIR /app COPY requirements.txt . RUN pip install -r requirements.txt COPY job.py . ENTRYPOINT ["python", "job.py"] ``

requirements.txt:

`` snowflake-snowpark-python==1.11.0 ``

Build and push the image as before.

Create and execute the job:

``sql EXECUTE JOB my_etl_job IN COMPUTE POOL my_pool FROM SPECIFICATION $$ spec: containers: - name: main image: /my_repo/my_job:latest args: ["MY_DB.MY_SCHEMA.SOURCE_TABLE", "MY_DB.MY_SCHEMA.RESULT_TABLE"] $$; ``

Monitor the job:

``sql SELECT * FROM TABLE(INFORMATION_SCHEMA.JOBS) WHERE NAME='MY_ETL_JOB'; CALL SYSTEM$GET_JOB_LOGS('MY_ETL_JOB'); ``

Jobs automatically terminate after completion. You can also schedule jobs using Snowflake tasks.

-- Execute a job
EXECUTE JOB my_etl_job
  IN COMPUTE POOL my_pool
  FROM SPECIFICATION $$
  spec:
    containers:
      - name: main
        image: /my_repo/my_job:latest
        args: ["MY_DB.MY_SCHEMA.SOURCE_TABLE", "MY_DB.MY_SCHEMA.RESULT_TABLE"]
  $$;

-- Check job status
SELECT * FROM TABLE(INFORMATION_SCHEMA.JOBS) WHERE NAME='MY_ETL_JOB';

-- Get logs
CALL SYSTEM$GET_JOB_LOGS('MY_ETL_JOB');

Building Interactive Apps with Streamlit

Snowpark Container Services also supports interactive web applications, such as Streamlit dashboards. Let’s deploy a simple app that queries Snowflake data.

Step 1: Create a Streamlit app

Create app.py:

```python import streamlit as st from snowflake.snowpark import Session

st.title("Snowflake Data Explorer")

session = Session.builder.configs({"connection_name": "snowflake_default"}).create()

table_name = st.text_input("Enter table name", "MY_DB.MY_SCHEMA.MY_TABLE")

if st.button("Query"): df = session.sql(f"SELECT * FROM {table_name} LIMIT 10").collect() st.dataframe(df)

session.close() ```

Step 2: Dockerfile

``dockerfile FROM python:3.9-slim WORKDIR /app COPY requirements.txt . RUN pip install -r requirements.txt COPY app.py . EXPOSE 8501 CMD ["streamlit", "run", "app.py", "--server.port=8501", "--server.enableCORS=false"] ``

requirements.txt:

`` streamlit==1.28.0 snowflake-snowpark-python==1.11.0 ``

Step 3: Build, push, and create the app

``sql CREATE APPLICATION my_app IN COMPUTE POOL my_pool FROM SPECIFICATION $$ spec: containers: - name: main image: /my_repo/my_app:latest endpoints: - name: app-endpoint port: 8501 public: true $$; ``

Step 4: Access the app

Get the endpoint URL from SHOW ENDPOINTS and open it in a browser.

Note: Apps are essentially services with a web UI. They can be shared with other users via Snowflake’s RBAC.

-- Create application
CREATE APPLICATION my_app
  IN COMPUTE POOL my_pool
  FROM SPECIFICATION $$
  spec:
    containers:
      - name: main
        image: /my_repo/my_app:latest
    endpoints:
      - name: app-endpoint
        port: 8501
        public: true
  $$;

-- Show endpoint
SHOW ENDPOINTS IN APPLICATION my_app;

Monitoring and Debugging Containers

Monitoring is crucial for production. Snowflake provides built-in functions to inspect containers.

Service Status

``sql CALL SYSTEM$GET_SERVICE_STATUS('my_service'); `` Returns JSON with instance states (ACTIVE, FAILED, etc.).

Service Logs

``sql CALL SYSTEM$GET_SERVICE_LOGS('my_service', 0, 'main'); `` Where 0 is the instance index and 'main' is the container name.

Job Logs

``sql CALL SYSTEM$GET_JOB_LOGS('my_job'); ``

Metrics

You can query Snowflake’s INFORMATION_SCHEMA for job history:

``sql SELECT * FROM TABLE(INFORMATION_SCHEMA.JOBS) WHERE START_TIME > DATEADD('hour', -24, CURRENT_TIMESTAMP); ``

Alerts

Set up Snowflake alerts on job failures or service downtimes using CREATE ALERT.

Common Issues - Container crash: Check logs for exceptions. Ensure all dependencies are installed. - Out of memory: Increase memory in the compute pool or optimize code. - Image pull failure: Verify image path and credentials. - Endpoint not reachable: Check that the port matches the container’s listening port and that the readiness probe passes.

-- Get service status
CALL SYSTEM$GET_SERVICE_STATUS('my_time_service');

-- Get logs for instance 0, container main
CALL SYSTEM$GET_SERVICE_LOGS('my_time_service', 0, 'main');

-- Get job logs
CALL SYSTEM$GET_JOB_LOGS('my_etl_job');

-- Query job history
SELECT * FROM TABLE(INFORMATION_SCHEMA.JOBS)
WHERE NAME = 'MY_ETL_JOB';

Security and Best Practices

Security is built-in, but you must follow best practices.

Authentication and Authorization - Use Snowflake’s RBAC to control who can create, manage, and access containers. - Grant USAGE on compute pools and image repositories to specific roles. - For services, use GRANT USAGE ON SERVICE to allow users to call endpoints.

Network Security - By default, containers have no network access to the internet. Use external access integrations to allow outbound connections. - For inbound, use private endpoints (public:false) and connect via Snowflake’s private link.

Secrets Management - Never hardcode credentials. Use Snowflake’s secrets (CREATE SECRET) and reference them in the spec via environment variables.

Resource Management - Set appropriate MIN/MAX_INSTANCES to control costs. - Use auto-suspend for compute pools (not yet available, but you can manually suspend).

Image Security - Scan images for vulnerabilities before pushing. - Use signed images to ensure integrity.

Example: Using a secret

```sql CREATE SECRET my_secret TYPE = PASSWORD USERNAME = 'user' PASSWORD = 'pass';

CREATE SERVICE my_service FROM SPECIFICATION $$ spec: containers: - name: main image: ... env: DB_PASSWORD: { "secret": "my_secret", "key": "password" } $$; ```

-- Create a secret
CREATE SECRET db_cred
  TYPE = PASSWORD
  USERNAME = 'admin'
  PASSWORD = 's3cret';

-- Grant usage
GRANT USAGE ON SECRET db_cred TO ROLE my_role;

-- Use in service spec
CREATE SERVICE secure_service
  FROM SPECIFICATION $$
  spec:
    containers:
      - name: main
        image: /my_repo/my_app:latest
        env:
          DB_PASSWORD: { "secret": "db_cred", "key": "password" }
  $$;