AWS Systems Manager: Operations Hub and SSM Agent

You're SSHing into 200 EC2 instances to apply a critical security patch. Halfway through, your session drops. You don't know which instances are patched, which are still vulnerable, and your audit trail is a mess of terminal logs. This is the reality of managing infrastructure without AWS Systems Manager. Systems Manager isn't just another AWS service — it's the operational backbone that eliminates the need for bastion hosts, SSH keys, and manual runbooks. By centralizing patch management, session management, and inventory tracking, it turns chaotic fleet operations into a repeatable, auditable process. If you're still managing instances with SSH and hoping for the best, you're doing it wrong.

What Is AWS Systems Manager and Why It Matters

AWS Systems Manager is the operational hub for managing EC2 instances, on-premises servers, and other AWS resources at scale. It centralizes operational data, automates routine tasks, and enforces compliance without requiring bastion hosts or SSH keys. The SSM Agent, installed on each managed instance, is the workhorse that executes commands, collects inventory, and applies patches. In production, Systems Manager replaces ad-hoc SSH access with auditable, role-based actions. It's not optional — if you manage more than a handful of instances, you need it to avoid configuration drift and security gaps. The key insight: Systems Manager turns instance management into API calls, enabling automation and reducing human error.

#!/bin/bash
# Install SSM Agent on Amazon Linux 2
sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
sudo systemctl enable amazon-ssm-agent
sudo systemctl start amazon-ssm-agent
# Verify agent is running
sudo systemctl status amazon-ssm-agent

SSM Agent Architecture and Communication Flow

The SSM Agent runs as a service on each instance and communicates with the Systems Manager service over HTTPS (outbound only). It uses an IAM role (instance profile) to authenticate and authorize actions. The agent polls the Systems Manager service every 5 seconds for pending commands, state changes, or session requests. This polling model means instances don't need inbound ports — a huge security win. The agent stores operational data locally in a SQLite database and sends it to Systems Manager for inventory, patch compliance, and execution history. Understanding this flow is critical: if the agent can't reach the service endpoint (ssm..amazonaws.com), all management stops. In production, we route traffic through VPC endpoints to avoid internet dependency and reduce latency.

{
  "settings": {
    "telemetry": {
      "enableTelemetry": true
    },
    "ssm": {
      "endpoint": "https://ssm.us-east-1.amazonaws.com",
      "healthCheckIntervalSeconds": 300
    },
    "agent": {
      "logLevel": "info",
      "maxLogLines": 10000
    }
  }
}

Setting Up SSM Agent with IAM Roles and Permissions

Every managed instance needs an IAM role (instance profile) with permissions to call Systems Manager APIs. The minimum policy is AmazonSSMManagedInstanceCore, which allows the agent to send heartbeats, receive commands, and upload logs. For advanced features like inventory or patch management, add policies like AmazonSSMInventoryFullAccess or AmazonSSMPatchAssociation. Never attach policies directly to instances — always use instance profiles. In production, we enforce least privilege by creating custom policies that restrict actions to specific resources (e.g., allow RunCommand only on tagged instances). A common mistake is forgetting to attach the instance profile to the EC2 instance at launch. You can attach it later, but the instance must be rebooted for the agent to pick up the new role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:DescribeAssociation",
        "ssm:GetDeployablePatchSnapshotForInstance",
        "ssm:GetDocument",
        "ssm:GetManifest",
        "ssm:GetParameters",
        "ssm:ListAssociations",
        "ssm:ListInstanceAssociations",
        "ssm:PutInventory",
        "ssm:PutComplianceItems",
        "ssm:PutConfigurePackageResult",
        "ssm:UpdateAssociationStatus",
        "ssm:UpdateInstanceAssociationStatus",
        "ssm:UpdateInstanceInformation"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstanceStatus"
      ],
      "Resource": "*"
    }
  ]
}

Operations Hub: Centralized Visibility and Automation

The Operations Hub is the Systems Manager console that aggregates operational data from all managed instances. It provides dashboards for inventory, patch compliance, session history, and Run Command execution. You can filter by tags, resource groups, or custom attributes. The real power is automation: you can create associations that automatically apply documents (e.g., patch baselines, inventory collections) on a schedule or on instance launch. In production, we use Operations Hub to enforce compliance: every instance must report inventory every 30 minutes and apply patches within 7 days. If an instance fails to report, we trigger a Lambda function to investigate. The hub also integrates with CloudWatch and AWS Config for a unified view. Without it, you're flying blind.

{
  "Name": "PatchBaselineAssociation",
  "DocumentName": "AWS-RunPatchBaseline",
  "Targets": [
    {
      "Key": "tag:Environment",
      "Values": ["production"]
    }
  ],
  "ScheduleExpression": "cron(0 2 ? * SUN *)",
  "Parameters": {
    "Operation": ["Install"]
  }
}

Run Command: Ad-Hoc and Automated Script Execution

Run Command lets you execute scripts or commands on one or many instances without SSH. You can run shell scripts, PowerShell, or SSM documents. Commands are executed via the SSM Agent, and output is streamed to CloudWatch Logs or S3. In production, we use Run Command for emergency patching, log collection, and configuration changes. The key advantage is auditability: every command is logged with who ran it, when, and the output. You can also use rate control to limit concurrency and error thresholds to prevent blast radius. A common pattern is to wrap Run Command in a Step Functions workflow for multi-step automation. Never use Run Command for long-running processes — it has a 30-minute timeout. For longer tasks, use State Manager associations.

# Send a command to check disk usage on all production instances
aws ssm send-command \
    --document-name "AWS-RunShellScript" \
    --targets "Key=tag:Environment,Values=production" \
    --parameters '{"commands":["df -h"]}' \
    --comment "Check disk usage" \
    --output-s3-bucket-name my-ssm-logs \
    --max-concurrency 10 \
    --max-errors 2

Session Manager: Secure Shell Access Without Bastions

Session Manager provides browser-based or CLI shell access to instances without SSH keys, bastion hosts, or open inbound ports. It uses the SSM Agent to create a secure tunnel over HTTPS. Sessions are logged to CloudTrail and optionally to S3 for audit. In production, we've eliminated all SSH bastions — Session Manager is more secure and easier to manage. You can restrict access using IAM policies (e.g., allow session only for specific instances or during business hours). A common pitfall is forgetting to attach the necessary IAM policy to users: they need ssm:StartSession and ssm:TerminateSession. Also, ensure the instance profile has ssm:UpdateInstanceInformation. For Windows, you need to enable PowerShell remoting. Session Manager is not just a convenience — it's a security best practice.

# Start a session on an instance by instance ID
aws ssm start-session --target i-1234567890abcdef0

# Or start a session using a tag (requires AWS CLI v2)
aws ssm start-session --target "tag:Name=web-server-01"

Patch Manager: Automated OS Patching at Scale

Patch Manager automates the process of patching OS-level vulnerabilities across your fleet. You define patch baselines (rules for which patches are approved automatically) and create associations to apply them on a schedule. In production, we use Patch Manager to enforce a 7-day patch window for critical vulnerabilities. The system reports compliance per instance, and you can generate reports via Systems Manager or export to Athena. A common mistake is not testing patches on a canary group first — we once had a kernel patch that broke our monitoring agent. Always use maintenance windows to control when patches are applied, and set up CloudWatch Events to notify on patch failures. Patch Manager also supports custom patch baselines for third-party applications like MySQL or Apache.

{
  "Name": "Production-Patch-Baseline",
  "OperatingSystem": "AMAZON_LINUX_2",
  "ApprovalRules": {
    "PatchRules": [
      {
        "PatchFilterGroup": {
          "PatchFilters": [
            {
              "Key": "CLASSIFICATION",
              "Values": ["Security"]
            },
            {
              "Key": "SEVERITY",
              "Values": ["Critical", "Important"]
            }
          ]
        },
        "ApproveAfterDays": 7
      }
    ]
  }
}

Inventory Manager: Collecting and Querying Instance Metadata

Inventory Manager collects metadata from instances — installed applications, network configuration, services, and custom data — and stores it in a centralized Systems Manager inventory. You can query inventory using the console, CLI, or integrate with Athena for complex analysis. In production, we use inventory to track software versions, detect unauthorized applications, and audit compliance. The agent collects inventory every 30 minutes by default, but you can customize the interval. A key limitation: inventory only captures data at collection time — it's not real-time. For real-time monitoring, use CloudWatch metrics. Also, inventory data is stored in the Systems Manager service, which has a 30-day retention for some data types. Export to S3 for long-term storage. We once discovered a rogue application running on 20 instances only after inventory flagged it.

{
  "Name": "InventoryCollection",
  "DocumentName": "AWS-GatherSoftwareInventory",
  "Targets": [
    {
      "Key": "tag:Environment",
      "Values": ["production"]
    }
  ],
  "ScheduleExpression": "rate(1 hour)",
  "Parameters": {
    "applications": ["Enabled"],
    "awsComponents": ["Enabled"],
    "networkConfig": ["Enabled"],
    "windowsRoles": ["Disabled"],
    "windowsUpdates": ["Disabled"]
  }
}

State Manager: Enforcing Desired Configuration

State Manager ensures instances are in a desired state by applying SSM documents on a schedule or on events. It's like a lightweight configuration management tool (similar to Chef or Ansible) but native to AWS. You define associations that specify a document, targets, and schedule. State Manager automatically runs the document on new instances that match the targets. In production, we use State Manager to enforce security configurations: disable root SSH access, install the CloudWatch agent, and apply CIS benchmarks. The key advantage is that it's idempotent — running the same association multiple times won't cause issues. However, State Manager does not handle complex dependencies or ordering. For that, use Step Functions. A common failure mode is when an association fails due to a missing parameter — always validate documents in a test environment first.

{
  "Name": "Enforce-CloudWatch-Agent",
  "DocumentName": "AWS-ConfigureAWSPackage",
  "Targets": [
    {
      "Key": "tag:Role",
      "Values": ["web"]
    }
  ],
  "ScheduleExpression": "cron(0 0 * * ? *)",
  "Parameters": {
    "action": ["Install"],
    "name": ["AmazonCloudWatchAgent"],
    "version": ["latest"]
  }
}

Automation: Runbooks for Incident Response and Remediation

Automation allows you to create runbooks (SSM documents) that automate common operational tasks like restarting services, replacing instances, or rolling back deployments. Runbooks can include multiple steps, conditional logic, and error handling. They integrate with other AWS services via AWS SDK calls. In production, we use Automation runbooks for incident response: when an instance fails health checks, a runbook automatically terminates it, launches a replacement, and updates the load balancer. The key is to design runbooks idempotently — running them multiple times should be safe. A common mistake is hardcoding instance IDs or regions. Use dynamic parameters and AWS Systems Manager Parameter Store for configuration. Automation is powerful but can be dangerous: always test runbooks in a sandbox account first.

{
  "description": "Replace unhealthy EC2 instance",
  "schemaVersion": "0.3",
  "assumeRole": "arn:aws:iam::123456789012:role/AutomationRole",
  "parameters": {
    "InstanceId": {
      "type": "String",
      "description": "ID of the unhealthy instance"
    },
    "AmiId": {
      "type": "String",
      "description": "AMI ID for replacement"
    }
  },
  "mainSteps": [
    {
      "name": "TerminateInstance",
      "action": "aws:executeAwsApi",
      "inputs": {
        "Service": "ec2",
        "Api": "TerminateInstances",
        "InstanceIds": ["{{ InstanceId }}"]
      }
    },
    {
      "name": "LaunchReplacement",
      "action": "aws:executeAwsApi",
      "inputs": {
        "Service": "ec2",
        "Api": "RunInstances",
        "ImageId": "{{ AmiId }}",
        "InstanceType": "t3.micro",
        "MinCount": 1,
        "MaxCount": 1
      }
    }
  ]
}

Monitoring and Troubleshooting SSM Agent and Operations

Monitoring SSM Agent health is critical. Use CloudWatch metrics (ssm-agent-status, ssm-agent-version) and set alarms for instances that stop reporting. Common issues: agent not running, network connectivity problems, IAM role misconfiguration, or disk space full. The agent logs are in /var/log/amazon/ssm/ on Linux and C:\ProgramData\Amazon\SSM\Logs on Windows. For troubleshooting, check the agent log for errors like 'Unable to connect to service' or 'Invalid instance profile'. Also, use the Systems Manager console's 'Instance Management' page to see the last ping time. If an instance shows 'Connection Lost', it's likely a network or IAM issue. In production, we run a scheduled Lambda that checks agent status and auto-remediates by restarting the agent or re-attaching the instance profile.

#!/bin/bash
# Check SSM Agent status and restart if not running
if ! systemctl is-active --quiet amazon-ssm-agent; then
    echo "SSM Agent is not running. Restarting..."
    sudo systemctl restart amazon-ssm-agent
    sleep 5
    if systemctl is-active --quiet amazon-ssm-agent; then
        echo "SSM Agent restarted successfully."
    else
        echo "Failed to restart SSM Agent."
        exit 1
    fi
else
    echo "SSM Agent is running."
fi

Production Best Practices and Common Pitfalls

After years of running Systems Manager in production, here are the non-negotiables: 1) Always use VPC endpoints for SSM, EC2, and S3 to avoid internet dependency. 2) Enforce least privilege IAM policies — never use wildcards for actions. 3) Test all associations and runbooks in a staging environment first. 4) Enable CloudTrail logging for all Systems Manager actions. 5) Set up CloudWatch alarms for agent health and patch compliance. 6) Use maintenance windows for patching and reboots. 7) Regularly audit inventory and compliance reports. Common pitfalls: forgetting to attach instance profiles, using default patch baselines without testing, and not handling agent failures gracefully. Also, be aware of service limits: you can have up to 5000 instances per account, but associations and documents have limits. Plan for scale by using resource groups and tagging strategies.

#!/bin/bash
# List instances not compliant with patch baseline
aws ssm describe-instance-patch-states \
    --instance-ids $(aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId' --output text) \
    --query 'InstancePatchStates[?PatchComplianceStatus!=`COMPLIANT`].[InstanceId, PatchComplianceStatus]' \
    --output table