Ansible Windows Automation: WinRM CredSSP Pitfall with Double-Hop & Fix

I still remember the outage. We had 200 Windows servers that needed a hotfix deployed via Ansible. The playbook worked perfectly on the first 50, then suddenly failed with 'WinRM transport failed' on a cluster of servers in a different domain. The error message was cryptic: 'The WinRM client cannot process the request because the server name is not in the trusted hosts list.' We had set TrustedHosts, but the issue was deeper — it was a Kerberos double-hop problem. That night, I learned the hard way that Windows automation with Ansible is not just about installing pywinrm and running a playbook. It's about understanding WinRM authentication protocols, Group Policy settings, and the subtle differences between win_shell and win_command. This article covers everything I wish I knew then.

1. WinRM Configuration on Windows Hosts

Before Ansible can manage a Windows host, you need to configure WinRM. The quickest way is to run Enable-PSRemoting -Force in an elevated PowerShell. This starts the WinRM service, sets it to auto-start, and creates HTTP and HTTPS listeners. However, for production, you need to secure it.

HTTPS Listener with Self-Signed Cert: ``powershell $cert = New-SelfSignedCertificate -CertStoreLocation Cert:\LocalMachine\My -DnsName $env:COMPUTERNAME -FriendlyName 'WinRM HTTPS' New-WSManInstance -ResourceURI winrm/config/Listener -SelectorSet @{Address='*';Transport='HTTPS'} -ValueSet @{CertificateThumbprint=$cert.Thumbprint} ``

Firewall rules: ``powershell New-NetFirewallRule -DisplayName 'WinRM HTTP' -Direction Inbound -Protocol TCP -LocalPort 5985 -Action Allow New-NetFirewallRule -DisplayName 'WinRM HTTPS' -Direction Inbound -Protocol TCP -LocalPort 5986 -Action Allow ``

TrustedHosts: For non-domain or cross-domain, set TrustedHosts to allow connections: ``powershell Set-Item WSMan:\localhost\Client\TrustedHosts -Value '*' -Force ``

Ansible Inventory for Windows: ``yaml [windows] server1 ansible_host=192.168.1.10 ansible_user=Administrator ansible_password=Passw0rd ansible_connection=winrm ansible_winrm_server_cert_validation=ignore ``

Production insight: I once spent hours debugging a connection issue only to find the WinRM service was set to manual start after a Windows update. Always verify with Get-Service WinRM.

---
- name: Configure WinRM on Windows hosts
  hosts: windows
  gather_facts: no
  tasks:
    - name: Enable WinRM and set max memory per shell
      ansible.windows.win_shell: |
        winrm set winrm/config/winrs @{MaxMemoryPerShellMB="2048"}
        winrm set winrm/config/winrs @{MaxShellsPerUser="30"}
      register: result
    - name: Set WinRM service to auto start and ensure running
      ansible.windows.win_service:
        name: WinRM
        start_mode: auto
        state: started
    - name: Allow WinRM through firewall
      ansible.windows.win_firewall_rule:
        name: WinRM HTTPS
        localport: 5986
        action: allow
        direction: in
        protocol: tcp
        enabled: yes

2. WinRM Authentication Protocols: NTLM, Kerberos, CredSSP

Ansible supports three main authentication transports: ntlm, kerberos, and credssp. Each has trade-offs.

NTLM: Default, works for workgroup and domain. Simple, but no double-hop. Use ansible_winrm_transport: ntlm.

Kerberos: Requires domain-joined hosts, time sync, and proper SPN. Set ansible_winrm_transport: kerberos. You need python-gssapi or pywinrm[kerberos] on the control node. Verify with klist get HTTP/target.domain.com.

CredSSP: Enables double-hop delegation. Set ansible_winrm_transport: credssp. On the target, run: ``powershell Enable-WSManCredSSP -Role Server -Force ` On the control node, install pip install pywinrm[credssp]`.

Choosing the right one: - Same domain, no double-hop: Kerberos (fastest) - Workgroup or cross-domain: NTLM - Need to access network shares: CredSSP

Production insight: We once had Kerberos fail because the DNS suffix was missing. Always use FQDN in ansible_host for Kerberos to work.

---
- name: Test WinRM authentication protocols
  hosts: windows
  gather_facts: no
  vars:
    ansible_connection: winrm
    ansible_winrm_transport: credssp
    ansible_winrm_server_cert_validation: ignore
  tasks:
    - name: Test NTLM authentication
      ansible.windows.win_ping:
      vars:
        ansible_winrm_transport: ntlm
      register: ping_ntlm
    - name: Test Kerberos authentication
      ansible.windows.win_ping:
      vars:
        ansible_winrm_transport: kerberos
      register: ping_kerberos
    - name: Test CredSSP authentication
      ansible.windows.win_ping:
      vars:
        ansible_winrm_transport: credssp
      register: ping_credssp
    - name: Show results
      ansible.builtin.debug:
        msg:
          - "NTLM: {{ ping_ntlm.ping }}"
          - "Kerberos: {{ ping_kerberos.ping }}"
          - "CredSSP: {{ ping_credssp.ping }}"

3. The ansible_connection: winrm Plugin Deep Dive

The winrm connection plugin is what makes Ansible talk to Windows. Key variables:

• ansible_winrm_transport: ntlm, kerberos, credssp, basic
• ansible_winrm_port: 5985 (HTTP) or 5986 (HTTPS)
• ansible_winrm_server_cert_validation: ignore or validate (default: validate)
• ansible_winrm_operation_timeout_sec: default 60
• ansible_winrm_read_timeout_sec: default 60
• ansible_winrm_pipeline_timeout_sec: default 60

Example inventory with advanced settings: ``yaml [windows:vars] ansible_connection=winrm ansible_winrm_transport=credssp ansible_winrm_port=5986 ansible_winrm_server_cert_validation=ignore ansible_winrm_operation_timeout_sec=120 ansible_winrm_read_timeout_sec=120 ansible_winrm_pipeline_timeout_sec=120 ``

Why timeouts matter: Long-running PowerShell scripts may timeout. For example, a win_shell that runs a 5-minute script needs pipeline_timeout > 300.

Production insight: We had a deployment that kept failing with 'WinRM operation timeout' because the default 60 seconds wasn't enough for a large package installation. Bumping to 300 fixed it.

---
- name: Deep dive into winrm connection plugin
  hosts: windows
  gather_facts: no
  vars:
    ansible_connection: winrm
    ansible_winrm_transport: credssp
    ansible_winrm_port: 5986
    ansible_winrm_server_cert_validation: ignore
    ansible_winrm_operation_timeout_sec: 60
    ansible_winrm_read_timeout_sec: 70
  tasks:
    - name: Gather Windows facts
      ansible.builtin.setup:
      register: facts
    - name: Display OS version
      ansible.builtin.debug:
        var: facts.ansible_facts.ansible_os_name

4. Using win_service for Service Management

The win_service module manages Windows services. Common parameters: - name: service name (e.g., Spooler) - state: started, stopped, restarted - start_mode: auto, manual, disabled

Example playbook: ``yaml - name: Ensure print spooler is running and set to auto win_service: name: Spooler state: started start_mode: auto ``

Production gotcha: Service names are case-insensitive but must match exactly. Use Get-Service to verify.

Idempotency: win_service is idempotent; it will only change state if needed.

Restart with dependencies: ``yaml - name: Restart service and its dependencies win_service: name: W3Svc state: restarted dependencies: - WAS ``

Production insight: We once tried to stop a service that was already stopped, and win_service reported 'changed' because the start_mode was different. Always check the module's return values.

---
- name: Manage Windows services
  hosts: windows
  gather_facts: no
  tasks:
    - name: Ensure Spooler service is running and auto-start
      ansible.windows.win_service:
        name: Spooler
        state: started
        start_mode: auto
    - name: Stop and disable Print Spooler
      ansible.windows.win_service:
        name: Spooler
        state: stopped
        start_mode: disabled
    - name: Get service status
      ansible.windows.win_service_info:
        name: Spooler
      register: service_info
    - name: Show service info
      ansible.builtin.debug:
        var: service_info.services[0].state

5. Installing Software with win_package

win_package installs or uninstalls software using MSI or EXE. Key parameters: - path: path to installer - product_id: GUID for idempotency (required for MSI) - state: present or absent - arguments: additional CLI arguments

Example: ``yaml - name: Install 7-Zip win_package: path: '\\fileserver\share\7z1900-x64.msi' product_id: '{23170F69-40C1-2702-1900-000001000000}' state: present ``

Double-hop issue: If the installer is on a network share, use CredSSP or copy it locally first.

Getting product_id: ``powershell Get-WmiObject Win32_Product | Select-Object Name, IdentifyingNumber ``

Production insight: We once had an MSI that required a reboot. win_package doesn't handle reboots; you need to use win_reboot module after installation.

---
- name: Install software with win_package
  hosts: windows
  gather_facts: no
  tasks:
    - name: Install 7-Zip
      ansible.windows.win_package:
        path: https://www.7-zip.org/a/7z1900-x64.msi
        product_id: 7-Zip
        arguments: /quiet /norestart
        state: present
    - name: Install Notepad++
      ansible.windows.win_package:
        path: https://github.com/notepad-plus-plus/notepad-plus-plus/releases/download/v8.4.7/npp.8.4.7.Installer.x64.exe
        product_id: Notepad++
        arguments: /S
        state: present

6. File Transfer with win_copy

win_copy copies files from the control node to the Windows host (or between Windows hosts if using remote_src). Key parameters: - src: local path - dest: remote path - remote_src: yes if src is on the remote host

Example: ``yaml - name: Copy configuration file win_copy: src: ./app.config dest: C:\Program Files\App\app.config ``

Performance: win_copy uses SMB under the hood. For large files, consider using win_get_url or a file server.

Production gotcha: The destination folder must exist. Use win_file to create it first.

Copy from network share (requires CredSSP): ``yaml - name: Copy from share win_copy: src: '\\fileserver\share\file.txt' dest: C:\temp\file.txt remote_src: yes ``

Production insight: We had a 2GB file that timed out. We split it into chunks and used multiple win_copy tasks.

---
- name: File transfer with win_copy
  hosts: windows
  gather_facts: no
  tasks:
    - name: Copy a file from control node to Windows host
      ansible.windows.win_copy:
        src: /path/to/local/file.txt
        dest: C:\temp\file.txt
    - name: Copy a directory recursively
      ansible.windows.win_copy:
        src: /path/to/local/dir/
        dest: C:\temp\dir\
        remote_src: no
    - name: Fetch a file from Windows host to control node
      ansible.builtin.fetch:
        src: C:\temp\file.txt
        dest: /path/to/local/
        flat: yes

7. Running Commands: win_command vs win_shell

Both modules execute commands, but they differ in how they process output.

win_command: Runs a command via cmd.exe. Does not support PowerShell pipelines. Best for simple executables. ``yaml - name: Run ipconfig win_command: ipconfig ``

win_shell: Runs via PowerShell. Supports pipelines, variables, and complex scripts. ``yaml - name: Get services with status win_shell: Get-Service | Where-Object {$_.Status -eq 'Running'} | Select-Object Name ``

Key differences: - win_command returns stdout as string; win_shell returns stdout as string but can also return structured data if you use ConvertTo-Json. - win_shell has a executable parameter to choose the shell (e.g., powershell.exe, cmd.exe). - win_command is faster for simple commands.

Production insight: We used win_command to run a legacy batch file that failed because it needed PowerShell. Switching to win_shell with executable: cmd.exe fixed it.

---
- name: win_command vs win_shell
  hosts: windows
  gather_facts: no
  tasks:
    - name: Run a command with win_command (no shell)
      ansible.windows.win_command:
        cmd: ipconfig
      register: cmd_result
    - name: Run a command with win_shell (shell features)
      ansible.windows.win_shell: |
        $computers = Get-Content C:\temp\computers.txt
        foreach ($comp in $computers) {
          Test-Connection -ComputerName $comp -Count 1
        }
      register: shell_result
    - name: Show results
      ansible.builtin.debug:
        msg:
          - "win_command stdout: {{ cmd_result.stdout_lines[:2] }}"
          - "win_shell stdout: {{ shell_result.stdout_lines[:2] }}"

8. PowerShell Scripts via win_shell

For complex automation, write PowerShell scripts and execute them with win_shell. Best practices:

• Use executable: powershell.exe explicitly.
• Set ansible_winrm_pipeline_timeout_sec high enough.
• Use $ErrorActionPreference = 'Stop' to catch errors.
• Return data as JSON for easy parsing.

Example playbook with inline script: ```yaml - name: Run custom script win_shell: | $ErrorActionPreference = 'Stop' $result = Get-Service | Select-Object Name, Status ConvertTo-Json $result args: executable: powershell.exe register: script_output

• debug:
• var: script_output.stdout
• ```

External script file: ``yaml - name: Run script from file win_shell: | C:\scripts\deploy.ps1 -Param1 value1 args: executable: powershell.exe ``

Production insight: We had a script that took 10 minutes. The default timeout caused failures. We set ansible_winrm_pipeline_timeout_sec=900.

---
- name: Execute PowerShell scripts via win_shell
  hosts: windows
  gather_facts: no
  tasks:
    - name: Run inline PowerShell script
      ansible.windows.win_shell: |
        $services = Get-Service | Where-Object {$_.Status -eq 'Running'}
        $services | ForEach-Object { $_.Name }
      register: running_services
    - name: Run a PowerShell script file
      ansible.windows.win_shell: |
        C:\scripts\deploy.ps1 -Environment Production
      args:
        executable: powershell.exe
    - name: Show running services
      ansible.builtin.debug:
        var: running_services.stdout_lines

9. Configuring Windows Hosts for Ansible at Scale

For large-scale deployment, automate the WinRM configuration using Group Policy or DSC.

Group Policy settings: - Computer Configuration -> Administrative Templates -> Windows Components -> Windows Remote Management (WinRM) -> WinRM Service - Allow remote server management through WinRM: Enabled - Allow unencrypted traffic: Disabled (use HTTPS) - Windows Firewall: Allow inbound WinRM ports

PowerShell DSC for WinRM: ``powershell Configuration WinRMConfig { Node localhost { Script EnableWinRM { SetScript = { Enable-PSRemoting -Force Set-Item WSMan:\localhost\Client\TrustedHosts -Value '*' -Force } TestScript = { (Get-Service WinRM).Status -eq 'Running' } GetScript = { @{Result = 'Check WinRM'} } } } } ``

Ansible playbook to bootstrap Windows hosts: ``yaml - hosts: all gather_facts: no tasks: - name: Enable WinRM win_shell: Enable-PSRemoting -Force - name: Set TrustedHosts win_shell: Set-Item WSMan:\localhost\Client\TrustedHosts -Value '' -Force - name: Create HTTPS listener win_shell: | $cert = New-SelfSignedCertificate -DnsName $env:COMPUTERNAME -CertStoreLocation Cert:\LocalMachine\My New-WSManInstance -ResourceURI winrm/config/Listener -SelectorSet @{Address='';Transport='HTTPS'} -ValueSet @{CertificateThumbprint=$cert.Thumbprint} ``

Production insight: We used Ansible to bootstrap itself: first run with NTLM over HTTP, then switch to Kerberos over HTTPS.

---
- name: Configure Windows hosts at scale
  hosts: windows
  gather_facts: no
  tasks:
    - name: Enable WinRM via Group Policy (simulated)
      ansible.windows.win_shell: |
        Set-Item -Path WSMan:\localhost\Client\TrustedHosts -Value '*' -Force
        Restart-Service WinRM
    - name: Set WinRM HTTPS listener
      ansible.windows.win_shell: |
        $cert = New-SelfSignedCertificate -DnsName $env:COMPUTERNAME -CertStoreLocation Cert:\LocalMachine\My
        New-WSManInstance -ResourceURI winrm/config/Listener -SelectorSet @{Address='*';Transport='HTTPS'} -ValueSet @{CertificateThumbprint=$cert.Thumbprint}
    - name: Open firewall port for WinRM HTTPS
      ansible.windows.win_firewall_rule:
        name: WinRM HTTPS
        localport: 5986
        action: allow
        direction: in
        protocol: tcp
        enabled: yes

10. Common WinRM Pitfalls and How to Avoid Them

Here are the most frequent issues I've encountered:

1. WinRM service not running: ``powershell Get-Service WinRM | Start-Service Set-Service WinRM -StartupType Automatic ``

2. Firewall blocking ports: Check with Test-NetConnection -ComputerName target -Port 5986.

3. Certificate validation failure: Set ansible_winrm_server_cert_validation: ignore for self-signed certs.

4. Authentication failure due to time skew: Kerberos requires time within 5 minutes. Use NTP.

5. Double-hop access denied: Use CredSSP or copy files locally first.

6. Module not found (e.g., win_package): Ensure you have the latest ansible.windows collection: ansible-galaxy collection install ansible.windows.

7. PowerShell execution policy: Set to RemoteSigned: ``powershell Set-ExecutionPolicy RemoteSigned -Scope LocalMachine -Force ``

Production insight: We once had a host where WinRM was configured but the user account was locked out. Always check account status.

---
- name: Common WinRM pitfalls and fixes
  hosts: windows
  gather_facts: no
  vars:
    ansible_winrm_transport: credssp
    ansible_winrm_server_cert_validation: ignore
  tasks:
    - name: Fix double-hop issue with CredSSP
      ansible.windows.win_shell: |
        Enable-WSManCredSSP -Role Client -DelegateComputer * -Force
        Set-Item -Path WSMan:\localhost\Client\Auth\CredSSP -Value $true
    - name: Increase WinRM timeout
      ansible.windows.win_shell: |
        Set-Item -Path WSMan:\localhost\MaxTimeoutms -Value 60000
        Set-Item -Path WSMan:\localhost\Shell\MaxMemoryPerShellMB -Value 2048
    - name: Test connection
      ansible.windows.win_ping:

11. Advanced: Using win_shell with PSSession for Complex Tasks

For tasks that require multiple commands in a single session (e.g., changing directory, setting variables), use a PSSession within win_shell.

Example: ``yaml - name: Run multi-step task win_shell: | $session = New-PSSession -ComputerName localhost Invoke-Command -Session $session -ScriptBlock { Set-Location C:\Deploy .\deploy.ps1 } Remove-PSSession $session ``

When to use: When you need to preserve state between commands (e.g., importing a module once, then using it).

Performance: Creating a PSSession adds overhead. Only use when necessary.

Production insight: We used this to run a script that required a specific working directory and environment variables.

---
- name: Advanced win_shell with PSSession
  hosts: windows
  gather_facts: no
  tasks:
    - name: Create PSSession to remote server
      ansible.windows.win_shell: |
        $session = New-PSSession -ComputerName ServerB -Credential (Get-Credential)
        Invoke-Command -Session $session -ScriptBlock {
          Get-Service -Name Spooler
        }
        Remove-PSSession $session
      register: pssession_result
    - name: Show result
      ansible.builtin.debug:
        var: pssession_result.stdout_lines

12. Monitoring and Logging Ansible Windows Automation

Enable detailed logging for troubleshooting.

Ansible logging: ``bash ANSIBLE_LOG_PATH=/var/log/ansible.log ansible-playbook playbook.yml -vvvv ``

WinRM logging on Windows: ``powershell Set-Item WSMan:\localhost\Client\MaxEnvelopeSizekb -Value 2048 Enable-WSManTrace ``

Collect WinRM logs: ``powershell Get-WinEvent -LogName Microsoft-Windows-WinRM/Operational | Select-Object -First 10 ``

Integration with monitoring: Use win_shell to send custom metrics to your monitoring system.

Production insight: We set up a centralized log collector to capture all Ansible runs and WinRM errors. This helped us identify a pattern of failures related to memory pressure.

---
- name: Monitoring and logging Ansible Windows automation
  hosts: windows
  gather_facts: no
  tasks:
    - name: Enable WinRM logging
      ansible.windows.win_shell: |
        wevtutil set-log Microsoft-Windows-WinRM/Operational /enabled:true /retention:true /maxsize:1073741824
    - name: Get recent WinRM logs
      ansible.windows.win_shell: |
        Get-WinEvent -LogName Microsoft-Windows-WinRM/Operational -MaxEvents 10 |
        Select-Object TimeCreated, Id, LevelDisplayName, Message |
        Format-Table -AutoSize
      register: winrm_logs
    - name: Show logs
      ansible.builtin.debug:
        var: winrm_logs.stdout_lines