Configure and Store Terraform Remote State in Azure Storage
They say you never store your eggs in one basket and this saying is true for your terraform state file. In this article Configuring Remote State for Terraform, I take you through the steps in creating a highly available Azure storage account which homes your remote state file and guarantees security and better collaboration. The following set of steps is what is needed to successfully achieve the creation and configuring of a remote state for your terraform projects.
So what is Terraform Remote State?
By default, Terraform stores state locally in a file named terraform.tfstate. When working with Terraform in a team, use of a local file makes Terraform usage complicated because each user must make sure they always have the latest state data before running Terraform and make sure that nobody else runs Terraform at the same time.
- Create resource group for Terraform state
- Create storage account
- Create Key Vault – If Non-Existent
- Export Storage Account Keys
- Create Storage Container
- Create Secret Key
- Store Storage Key in a Key-Vault
Store values in variables
RESOURCE_GROUP_NAME=remote-terraform-state STORAGE_ACCOUNT_NAME=tfstoragetrainingenc CONTAINER_NAME=remote-terraform-container KEYVAULT=tfkeyremotestate KEYVAULTSECRET=tfbackend-state-secret
Create resource group for Terraform state
# Create resource group for Terraform state az group create --name $RESOURCE_GROUP_NAME --location eastus
Create storage account
# Create storage account az storage account create --resource-group $RESOURCE_GROUP_NAME --name $STORAGE_ACCOUNT_NAME --sku Standard_LRS --encryption-services blob
Create Key Vault
#Create Key Vault az keyvault create --name $KEYVAULT --resource-group $RESOURCE_GROUP_NAME --location eastus
Export Storage Account Keys
#Export Storage Account Keys ACCOUNT_KEY=$(az storage account keys list --resource-group $RESOURCE_GROUP_NAME --account-name $STORAGE_ACCOUNT_NAME --query '[0].value' -o tsv) export ARM_ACCESS_KEY=$ACCOUNT_KEY
Create Storage Container
#Create Storage Container az storage container create --name $CONTAINER_NAME --account-name $STORAGE_ACCOUNT_NAME
Create Secret Key
#Create Secret Key az keyvault secret set --name $KEYVAULTSECRET --vault-name $KEYVAULT --value tfstatesecret
Store Storage Key in a Key-Vault
#Store Storage Key in a Key-Vault export ARM_ACCESS_KEY=$(az keyvault secret show --name $KEYVAULTSECRET --vault-name $KEYVAULT --query value -o tsv)
Verify the created keyvault resource
Verify the created resource
Terraform backend configuration in your providers.tf file
Terraform backend configuration sample – use unique names for your own configuration.
#IaC on Azure Cloud Platform | Declare Azure as the Provider
# Configure the Microsoft Azure Provider
terraform {
required_version = ">=0.12"
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "~>2.0"
}
}
#Azurerm Backend Configuration
backend "azurerm" {
resource_group_name = "remote-terraform-state"
storage_account_name = "tfstoragetrainingenc"
container_name = "remote-terraform-container"
key = "terraform.tfstate"
}
}
provider "azurerm" {
features {}
}
Create SSH Key
ssh-keygen -t rsa 4096 -f remotekey
Initialise Terraform to use the Remote State | Backend
terraform init Initializing the backend... Do you want to copy existing state to the new backend? Pre-existing state was found while migrating the previous "local" backend to the newly configured "azurerm" backend. No existing state was found in the newly configured "azurerm" backend. Do you want to copy this state to the new "azurerm" backend? Enter "yes" to copy and "no" to start with an empty state. Enter a value: yes
Once the backend configuration has been placed in the provider.tf configuration, you start to see the migration away from the local state to the azurerm backend.
PS C:\Workdir\terraform\azterraform> terraform init Initializing the backend... Do you want to copy existing state to the new backend? Pre-existing state was found while migrating the previous "local" backend to the newly configured "azurerm" backend. An existing non-empty state already exists in the new backend. The two states have been saved to temporary files that will be removed after responding to this query. Previous (type "local"): C:\Users\CLOUDA~1\AppData\Local\Temp\terraform2362619277\1-local.tfstate New (type "azurerm"): C:\Users\CLOUDA~1\AppData\Local\Temp\terraform2362619277\2-azurerm.tfstate Do you want to overwrite the state in the new backend with the previous state? Enter "yes" to copy and "no" to start with the existing state in the newly configured "azurerm" backend.
Newly created terraform state file showing unpopulated state
Selecting yes ensures that the existing terraform state file is now copied to the backend (remote state file).
Successfully configured the backend "azurerm"! Terraform will automatically use this backend unless the backend configuration changes. Initializing provider plugins... - Reusing previous version of hashicorp/template from the dependency lock file - Reusing previous version of hashicorp/azurerm from the dependency lock file - Reusing previous version of hashicorp/random from the dependency lock file - Reusing previous version of hashicorp/tls from the dependency lock file - Using previously-installed hashicorp/random v3.2.0 - Using previously-installed hashicorp/tls v3.4.0 - Using previously-installed hashicorp/template v2.2.0 - Using previously-installed hashicorp/azurerm v2.99.0 Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.
Pre-existing state was found while migrating the previous “local” backend to the newly configured “azurerm” backend.
Newly created terraform state file showing populated state
Verify the state of a resource – terraform state show
PS C:\Workdir\terraform\azterraform> terraform state show azurerm_resource_group.emc-eus2-corporate-import-rg
# azurerm_resource_group.emc-eus2-corporate-import-rg:
resource "azurerm_resource_group" "emc-eus2-corporate-import-rg" {
id = "/subscriptions/31e9c06e-6d3f-4485-836c-ff36c38135a3/resourceGroups/emc-eus2-corporate-import-rg"
location = "eastus2"
name = "emc-eus2-corporate-import-rg"
tags = {
"env" = "resource-group"
}
}
You can also create your remote state with HCL as per the configuration below in a terraform file like create-remote-storage.tf.
resource "random_string" "resource_code" {
length = 5
special = false
upper = false
}
resource "azurerm_resource_group" "tfstate" {
name = "tfstate"
location = "East US"
}
resource "azurerm_storage_account" "tfstate" {
name = "tfstate${random_string.resource_code.result}"
resource_group_name = azurerm_resource_group.tfstate.name
location = azurerm_resource_group.tfstate.location
account_tier = "Standard"
account_replication_type = "LRS"
allow_blob_public_access = true
tags = {
environment = "staging"
}
}
resource "azurerm_storage_container" "tfstate" {
name = "tfstate"
storage_account_name = azurerm_storage_account.tfstate.name
container_access_type = "blob"
}
Below are useful Terraform state commands
terraform state pull
This command downloads the state from its current location, upgrades the local copy to the latest state file version that is compatible with locally-installed Terraform, and outputs the raw format to stdout.
terraform state pull | sc terraform.tfstate
Troubleshooting Remote State Locks
╷ │ Error: Error acquiring the state lock │ │ Error message: state blob is already locked │ Lock Info: │ ID: 42ff26c4-3103-e54c-e21a-8a37a25e5884 │ Path: tfbackendcontainer/terraform.tfstate │ Operation: OperationTypeApply │ Who: DESKTOP-ATCRUJV\Cloud Architect@DESKTOP-ATCRUJV │ Version: 1.1.9 │ Created: 2022-06-06 22:53:18.7745943 +0000 UTC │ Info: │ │ │ Terraform acquires a state lock to protect the state from being written │ by multiple users at the same time. Please resolve the issue above and try │ again. For most commands, you can disable locking with the "-lock=false" │ flag, but this is not recommended.
terraform destroy -lock=false
│ Error: Failed to save state │ │ Error saving state: blobs.Client#PutBlockBlob: Failure responding to request: StatusCode=412 -- Original Error: autorest/azure: Service returned an error. Status=412 Code="LeaseIdMissing" Message="There is currently a │ lease on the blob and no lease ID was specified in the request.\nRequestId:110febce-c01e-000b-1326-7a1b95000000\nTime:2022-06-07T04:27:53.9123101Z" ╵
╷ │ Error: Failed to persist state to backend │ │ The error shown above has prevented Terraform from writing the updated state to the configured backend. To allow for recovery, the state has been written to the file "errored.tfstate" in the current working directory. │ │ Running "terraform apply" again at this point will create a forked state, making it harder to recover. │ │ To retry writing this state, use the following command: │ terraform state push errored.tfstate
The lock has occurred because there was a terraform action initiated by a user which has placed that lock. To fix it, go to Azure and on the blob break the lease and delete the lock.
terraform destroy -lock=false