Microsoft Azure — Blob Storage & Lifecycle

Azure is Microsoft's cloud computing platform offering over 200 services. This article covers blob storage & lifecycle with production-ready configurations, best practices, and hands-on examples.

Why Blob Lifecycle Management Matters in Production

In production, blob storage costs can spiral out of control if you treat all data equally. Hot data (accessed frequently) belongs in hot tier; cold data (accessed quarterly) belongs in cool or archive tiers. Without lifecycle policies, you pay premium for data that hasn't been touched in months. Worse, you risk compliance violations if you delete data too early or retain it too long. Azure Blob Lifecycle Management automates tier transitions and deletions based on age, last access, or tags. This isn't a 'nice to have' — it's a cost-control and governance necessity. I've seen teams burn $50k/month on hot storage for logs they never read. A simple lifecycle rule moved those logs to cool after 30 days and archive after 90, cutting costs by 70%. The key is to define your data classes upfront: transient, warm, cold, frozen. Each class maps to a tier and a retention policy. Lifecycle policies apply at the storage account level but can target specific containers or blob types via filters. They run once per day, so don't expect instant transitions. Plan for a 24-hour lag. Also, note that lifecycle actions are irreversible for deletion — test on a non-production account first. Finally, monitor policy runs via Azure Monitor metrics to catch failures (e.g., insufficient permissions on target tier).

{
  "rules": [
    {
      "name": "Move logs to cool after 30 days",
      "enabled": true,
      "type": "Lifecycle",
      "definition": {
        "filters": {
          "blobTypes": ["blockBlob"],
          "prefixMatch": ["logs/"]
        },
        "actions": {
          "baseBlob": {
            "tierToCool": {
              "daysAfterModificationGreaterThan": 30
            },
            "tierToArchive": {
              "daysAfterModificationGreaterThan": 90
            },
            "delete": {
              "daysAfterModificationGreaterThan": 365
            }
          }
        }
      }
    }
  ]
}

Tiering Strategies: Hot, Cool, Cold, and Archive

Azure Blob Storage offers four access tiers: Hot (low latency, high cost), Cool (lower cost, slightly higher latency), Cold (even lower cost, 30-day minimum), and Archive (cheapest, but rehydration takes hours). Choosing the right tier for each data class is critical. For example, user-uploaded profile pictures should stay in Hot for fast access. Application logs can move to Cool after 30 days, then to Cold after 90, and Archive after a year. But beware: Archive has a 180-day minimum storage charge — if you delete before that, you pay the remainder. Also, rehydrating from Archive to Hot costs both time and money (read operations + early deletion fees). In production, I recommend using Cool as the default tier for new blobs, then let lifecycle rules promote to Hot only if accessed frequently (using last access time tracking). Enable 'last access time tracking' on the storage account to base policies on actual usage, not just creation date. This prevents 'zombie data' — blobs that are never read but stay in Hot because they were created recently. Also, consider using 'blob index tags' for fine-grained filtering. For instance, tag blobs with 'retention=7days' and delete them after 7 days. This is more flexible than prefix matching. Remember: tier transitions are free from Hot to Cool, but Cool to Archive costs write operations. Plan your data flow to minimize costs.

# Enable last access time tracking
$storageAccount = Get-AzStorageAccount -ResourceGroupName "myResourceGroup" -Name "mystorageaccount"
$storageAccount.LastAccessTimeTrackingPolicy = @{
    Enable = $true
    Name = "default"
    TrackingGranularityInDays = 1
}
Set-AzStorageAccount -InputObject $storageAccount

Writing Lifecycle Policies with ARM Templates

Infrastructure as Code (IaC) is non-negotiable for production. You should define lifecycle policies in ARM templates or Bicep, not click-ops. This ensures consistency across environments and enables code review. An ARM template for lifecycle management includes a 'Microsoft.Storage/storageAccounts/managementPolicies' resource. The policy JSON is embedded as the 'policy' property. You can use parameters for rule names, thresholds, and filters to reuse across dev, staging, and prod. One pitfall: ARM template deployments are idempotent, but if you remove a rule from the template, it will be deleted on next deployment. To avoid accidental deletion, use incremental deployment mode and keep rules in the template even if disabled. Also, lifecycle policies are regional — if you have geo-replicated storage, the policy applies only to the primary region. For secondary, you need a separate policy. Another tip: use 'dependsOn' to ensure the storage account exists before applying the policy. And always validate the ARM template with 'Test-AzResourceGroupDeployment' before deploying. I once saw a team deploy a policy that accidentally targeted all blobs (no prefix filter) and moved everything to Archive, causing a 2-day outage while rehydrating. Always include a prefix filter or blob index tag filter to scope the policy.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "storageAccountName": {
      "type": "string"
    },
    "coolDays": {
      "type": "int",
      "defaultValue": 30
    },
    "archiveDays": {
      "type": "int",
      "defaultValue": 90
    },
    "deleteDays": {
      "type": "int",
      "defaultValue": 365
    }
  },
  "resources": [
    {
      "type": "Microsoft.Storage/storageAccounts/managementPolicies",
      "apiVersion": "2023-01-01",
      "name": "[concat(parameters('storageAccountName'), '/default')]",
      "properties": {
        "policy": {
          "rules": [
            {
              "name": "log-lifecycle",
              "enabled": true,
              "type": "Lifecycle",
              "definition": {
                "filters": {
                  "blobTypes": ["blockBlob"],
                  "prefixMatch": ["logs/"]
                },
                "actions": {
                  "baseBlob": {
                    "tierToCool": {
                      "daysAfterModificationGreaterThan": "[parameters('coolDays')]"
                    },
                    "tierToArchive": {
                      "daysAfterModificationGreaterThan": "[parameters('archiveDays')]"
                    },
                    "delete": {
                      "daysAfterModificationGreaterThan": "[parameters('deleteDays')]"
                    }
                  }
                }
              }
            }
          ]
        }
      }
    }
  ]
}

Automating Lifecycle with Azure CLI and PowerShell

For ad-hoc changes or CI/CD pipelines, Azure CLI and PowerShell are essential. You can create, update, or delete lifecycle policies without full ARM deployments. For example, to add a rule to move backups to Cool after 7 days: 'az storage account management-policy create --account-name mystorageaccount --resource-group myResourceGroup --policy @policy.json'. But be careful: this command replaces the entire policy. To update a single rule, you must get the current policy, modify it, and set it again. A better approach is to use 'az storage account management-policy update' with a JSON file containing only the rules you want. In PowerShell, use 'Set-AzStorageAccountManagementPolicy' with a policy object. I recommend storing the policy JSON in a Git repo and using a pipeline to deploy it. For example, an Azure DevOps pipeline can run a script that downloads the policy from a secure location and applies it. Also, use '--dry-run' flag to preview changes before applying. This is especially useful in CI/CD to catch errors early. Another tip: use 'az storage account management-policy show' to export the current policy as a baseline. Then diff against your desired policy to see what changed. This helps during audits.

#!/bin/bash
# Apply lifecycle policy from JSON file
ACCOUNT_NAME="mystorageaccount"
RESOURCE_GROUP="myResourceGroup"
POLICY_FILE="lifecycle-policy.json"

# Dry run first
echo "Dry run..."
az storage account management-policy create \
    --account-name $ACCOUNT_NAME \
    --resource-group $RESOURCE_GROUP \
    --policy @$POLICY_FILE \
    --dry-run

# Apply if dry run succeeds
echo "Applying policy..."
az storage account management-policy create \
    --account-name $ACCOUNT_NAME \
    --resource-group $RESOURCE_GROUP \
    --policy @$POLICY_FILE

Monitoring Lifecycle Policy Execution

Lifecycle policies run once per day, but you need to know if they executed successfully. Azure provides metrics: 'BlobCount' and 'BlobCapacity' by tier, and 'LifecyclePolicyRun' logs in Azure Monitor. Set up alerts for policy failures. For example, if a rule fails to move blobs to Archive because the target tier is disabled, you'll see 'LifecyclePolicyRun' with status 'Failed'. Also, track the number of blobs transitioned per day using 'BlobCount' metric split by tier. If the count doesn't change, the policy might not be matching any blobs. Another key metric: 'BlobCapacity' by tier — if Archive capacity doesn't increase, your policy isn't working. I recommend creating a dashboard with these metrics. Also, enable diagnostic settings for the storage account to send logs to Log Analytics. Then query: 'StorageLifecyclePolicyRunLogs | where Status == "Failed"'. This helps you catch issues like 'Insufficient permissions' or 'Blob locked by lease'. In production, I've seen policies fail because a blob had an infinite lease (e.g., from a VM). Lifecycle can't delete or move leased blobs. You need to break the lease first or exclude those blobs via filter. Finally, set up a weekly report that shows the cost savings from tier transitions. This helps justify the effort to management.

StorageLifecyclePolicyRunLogs
| where TimeGenerated > ago(7d)
| where Status == "Failed"
| project TimeGenerated, RuleName, BlobUri, FailureReason, Status
| order by TimeGenerated desc

Handling Blob Rehydration from Archive

Rehydrating blobs from Archive is expensive and slow. You have two priorities: Standard (up to 15 hours) and High (under 1 hour, but costs more). In production, you should design your application to handle rehydration delays. For example, if a user requests an archived blob, return a 202 Accepted with a location header pointing to a rehydration status endpoint. Then poll until the blob is available. Use 'Set-AzStorageBlobRehydrationPriority' to set priority. Also, consider using 'Blob Batch' to rehydrate multiple blobs at once. But beware: rehydration creates a copy of the blob in the target tier, so you'll have two copies until you delete the original. This can double storage costs temporarily. Another approach: use 'Blob Snapshots' or 'Versioning' to keep a hot copy of frequently accessed data while archiving the base blob. But this adds complexity. In practice, I recommend keeping a small hot cache (e.g., Azure CDN or Redis) for the most recent archives. Also, set up alerts for rehydration failures. Common failures: insufficient permissions, target tier full, or blob deleted during rehydration. Finally, test rehydration with a sample blob before going live. I once saw a team rehydrate 10 TB of data only to find the application couldn't handle the latency — they had to re-architect.

# Rehydrate a blob from Archive to Cool
$storageAccount = Get-AzStorageAccount -ResourceGroupName "myResourceGroup" -Name "mystorageaccount"
$ctx = $storageAccount.Context
$containerName = "mycontainer"
$blobName = "myblob.log"

# Set rehydration priority to High
Set-AzStorageBlobRehydrationPriority -Context $ctx -Container $containerName -Blob $blobName -RehydrationPriority High

# Then copy the blob to Cool tier
Start-AzStorageBlobCopy -Context $ctx -SrcContainer $containerName -SrcBlob $blobName -DestContainer $containerName -DestBlob $blobName -DestContext $ctx -StandardBlobTier Cool

# Monitor copy status
Get-AzStorageBlobCopyState -Context $ctx -Container $containerName -Blob $blobName

Cost Optimization with Lifecycle Policies

The primary goal of lifecycle management is cost savings. But you need to measure it. Use Azure Cost Management to track storage costs by tier. Set budgets and alerts. For example, if Archive costs spike, it might indicate a policy is moving too much data too early. Also, consider the cost of lifecycle operations themselves: each tier transition is a write operation (except Hot to Cool, which is free). So moving millions of small blobs can incur significant write costs. In production, batch small blobs into larger ones (e.g., 1 MB) to reduce operation counts. Another cost trap: early deletion fees. Archive has a 180-day minimum; Cool has 30-day minimum. If you delete before that, you pay the remaining days. So don't move data to Archive if you might need to delete it within 6 months. Use Cool or Cold instead. Also, use 'Blob Inventory' to analyze your data and identify candidates for tiering. For example, find blobs not accessed in 90 days and move them to Cool. Finally, consider using 'Azure Storage Actions' (preview) for more complex automation, like moving blobs based on tags. But lifecycle policies are simpler and sufficient for most cases.

{
  "rules": [
    {
      "name": "Move cold data to Archive after 180 days",
      "enabled": true,
      "type": "Lifecycle",
      "definition": {
        "filters": {
          "blobTypes": ["blockBlob"],
          "prefixMatch": ["cold-data/"]
        },
        "actions": {
          "baseBlob": {
            "tierToArchive": {
              "daysAfterModificationGreaterThan": 180
            }
          }
        }
      }
    }
  ]
}

Lifecycle Policies with Blob Versioning and Snapshots

If you enable blob versioning or snapshots, lifecycle policies can also manage them. For example, you can delete old versions after 30 days. This is crucial for cost control because each version is a separate blob. Without lifecycle, versioning can explode storage costs. Use 'actions.baseBlob' for current versions and 'actions.snapshot' for snapshots. For versions, use 'actions.version'. Note: deleting a version is permanent — you can't recover it. So set conservative thresholds. In production, I recommend keeping only the last N versions (e.g., 5) and deleting older ones. But lifecycle doesn't support 'keep last N' — you must use age-based rules. Alternatively, use 'Blob Inventory' to identify versions to delete and run a script. Another consideration: soft delete. If you enable soft delete, lifecycle policies run before soft delete, so deleted blobs go to soft delete first. This adds a layer of protection. But be careful: soft delete retention period should be shorter than lifecycle delete to avoid conflicts. For example, if soft delete retains for 7 days and lifecycle deletes after 30 days, the blob will be soft-deleted after 30 days and then permanently deleted after 37 days. This can be confusing. I recommend setting soft delete retention to 0 if you use lifecycle for deletion, or ensure they are aligned.

{
  "rules": [
    {
      "name": "Delete old versions after 30 days",
      "enabled": true,
      "type": "Lifecycle",
      "definition": {
        "filters": {
          "blobTypes": ["blockBlob"]
        },
        "actions": {
          "version": {
            "delete": {
              "daysAfterCreationGreaterThan": 30
            }
          }
        }
      }
    }
  ]
}

Common Pitfalls and How to Avoid Them

Lifecycle policies seem simple but have many gotchas. First, policy evaluation is based on blob modification time, not creation time. If you modify a blob, the clock resets. So a blob that is frequently updated may never move to Cool. Use last access time if that's a problem. Second, lifecycle policies don't apply to blobs in the 'Archive' tier — they are already there. To move them further, you need to rehydrate first. Third, policies are evaluated once per day, so don't expect immediate transitions. Fourth, if you have a large number of blobs (millions), the policy run may take longer than 24 hours, causing overlaps. Azure handles this by queuing runs, but you might see delays. Fifth, filters are case-sensitive. 'logs/' matches 'logs/foo' but not 'Logs/foo'. Use consistent naming. Sixth, lifecycle policies can't move blobs across storage accounts. If you need to move data to a different account, use AzCopy or Azure Data Factory. Seventh, if you delete a container, the lifecycle policy for blobs inside it becomes orphaned — it won't cause errors but won't do anything. Finally, always test with a small set of blobs before rolling out to production. Use a separate storage account for testing.

#!/bin/bash
# Create test blobs and verify lifecycle
ACCOUNT_NAME="teststorage"
RESOURCE_GROUP="testRG"
CONTAINER="test-container"

# Create container
az storage container create --account-name $ACCOUNT_NAME --name $CONTAINER

# Upload a blob with old modification time
az storage blob upload --account-name $ACCOUNT_NAME --container-name $CONTAINER --name test-blob.log --file /dev/null --content-type "text/plain"
# Set last modified to 60 days ago (requires Azure CLI with --if-modified-since? Use workaround: copy with metadata)
# Actually, easier: use PowerShell to set blob properties

# Apply a test policy that moves to Cool after 1 day
az storage account management-policy create --account-name $ACCOUNT_NAME --resource-group $RESOURCE_GROUP --policy @test-policy.json

# Wait for next run (or force with REST API? Not possible)
echo "Policy applied. Wait 24 hours for evaluation."

Integrating Lifecycle with Azure Policy and Governance

For enterprise environments, you need to enforce lifecycle policies across all storage accounts. Use Azure Policy to audit or enforce that a lifecycle policy exists. For example, create a policy that requires a lifecycle rule for all storage accounts with a specific tag. If an account doesn't have a policy, mark it as non-compliant. You can also use Azure Policy to set default lifecycle rules. For instance, deploy a policy that adds a rule to move logs to Cool after 30 days. This ensures consistency. Another governance aspect: data classification. Use blob index tags to mark data as 'PII', 'public', etc. Then lifecycle policies can act on tags. For example, delete PII data after 90 days for compliance. Azure Policy can also enforce that certain tags are present. In production, I recommend a three-tier governance: 1) Azure Policy to mandate lifecycle policies, 2) Azure Blueprints to deploy standard policies, 3) Azure Monitor to alert on non-compliance. Also, use 'Activity Log' to track changes to lifecycle policies. Who changed the policy? When? This is crucial for audits. Finally, consider using 'Azure Resource Graph' to query all storage accounts and their lifecycle policies. This gives you a bird's-eye view.

{
  "properties": {
    "displayName": "Require lifecycle policy on storage accounts",
    "policyType": "Custom",
    "mode": "All",
    "parameters": {},
    "policyRule": {
      "if": {
        "allOf": [
          {
            "field": "type",
            "equals": "Microsoft.Storage/storageAccounts"
          },
          {
            "field": "Microsoft.Storage/storageAccounts/managementPolicies[*].policy.rules[*].name",
            "exists": "false"
          }
        ]
      },
      "then": {
        "effect": "audit"
      }
    }
  }
}

Real-World Production Scenario: Log Management

Let's walk through a real scenario: managing application logs. You have a microservices architecture generating 500 GB of logs per day. Logs are stored in a container per service. Requirements: keep logs in Hot for 7 days for real-time debugging, move to Cool for 30 days for occasional analysis, move to Cold for 90 days for compliance, then Archive for 1 year, and delete after 2 years. Also, some logs contain PII and must be deleted after 90 days. Solution: Use blob index tags to mark PII logs. Create two lifecycle rules: one for non-PII logs (prefix 'logs/') with the tiering schedule, and one for PII logs (tag 'PII=true') that deletes after 90 days. Enable last access time tracking to ensure logs that are read frequently stay in Hot. Also, enable versioning to keep a history of log files (in case of tampering), but delete versions after 30 days. Monitor policy runs daily. Set up alerts for failures. Use Azure Policy to enforce that all storage accounts in the subscription have a lifecycle policy. This scenario saved a company $200k/year in storage costs. The key was the PII rule — without it, they would have violated GDPR.

{
  "rules": [
    {
      "name": "Non-PII logs tiering",
      "enabled": true,
      "type": "Lifecycle",
      "definition": {
        "filters": {
          "blobTypes": ["blockBlob"],
          "prefixMatch": ["logs/"],
          "blobIndexMatch": [
            {
              "name": "PII",
              "op": "!=",
              "value": "true"
            }
          ]
        },
        "actions": {
          "baseBlob": {
            "tierToCool": {
              "daysAfterModificationGreaterThan": 7
            },
            "tierToCold": {
              "daysAfterModificationGreaterThan": 30
            },
            "tierToArchive": {
              "daysAfterModificationGreaterThan": 90
            },
            "delete": {
              "daysAfterModificationGreaterThan": 730
            }
          }
        }
      }
    },
    {
      "name": "PII logs deletion",
      "enabled": true,
      "type": "Lifecycle",
      "definition": {
        "filters": {
          "blobTypes": ["blockBlob"],
          "blobIndexMatch": [
            {
              "name": "PII",
              "op": "==",
              "value": "true"
            }
          ]
        },
        "actions": {
          "baseBlob": {
            "delete": {
              "daysAfterModificationGreaterThan": 90
            }
          }
        }
      }
    }
  ]
}

Future-Proofing: Lifecycle with Azure Storage Actions

Azure Storage Actions (preview) extend lifecycle capabilities. They allow you to run custom logic on blobs, like moving blobs based on complex conditions (e.g., file size > 1 GB). They are event-driven and can run on a schedule. However, they are more expensive than lifecycle policies. For most use cases, lifecycle policies suffice. But if you need conditional logic beyond age/tags, consider Storage Actions. For example, move blobs larger than 100 MB to Cool immediately. Or delete blobs that haven't been accessed in 90 days AND are larger than 1 GB. Storage Actions use a JSON-based definition and can be triggered by blob events. They also support idempotency. In production, I recommend using lifecycle policies as the first line of defense, and Storage Actions for edge cases. Also, keep an eye on Azure's roadmap — lifecycle policies may gain more features. For now, the combination of lifecycle policies + blob index tags + inventory reports covers 95% of scenarios.

{
  "properties": {
    "description": "Move large blobs to Cool",
    "triggers": {
      "type": "Schedule",
      "schedule": {
        "frequency": "Day",
        "interval": 1
      }
    },
    "actions": [
      {
        "name": "tierToCool",
        "if": {
          "condition": "blobSize > 104857600"
        },
        "then": {
          "setBlobTier": "Cool"
        }
      }
    ]
  }
}