They say you never store your eggs in one basket and this saying is true for your terraform state file. In this article Configuring Remote State for Terraform, I take you through the steps in creating a highly available Azure storage account which homes your remote state file and guarantees security and better collaboration. The following set of steps is what is needed to successfully achieve the creation and configuring of a remote state for your terraform projects.

So what is Terraform Remote State?

By default, Terraform stores state locally in a file named terraform.tfstate. When working with Terraform in a team, use of a local file makes Terraform usage complicated because each user must make sure they always have the latest state data before running Terraform and make sure that nobody else runs Terraform at the same time.

1. Create resource group for Terraform state
2. Create storage account
3. Create Key Vault – If Non-Existent
4. Export Storage Account Keys
5. Create Storage Container
6. Create Secret Key
7. Store Storage Key in a Key-Vault

Store values in variables

RESOURCE_GROUP_NAME=remote-terraform-state
STORAGE_ACCOUNT_NAME=tfstoragetrainingenc
CONTAINER_NAME=remote-terraform-container
KEYVAULT=tfkeyremotestate
KEYVAULTSECRET=tfbackend-state-secret

Create resource group for Terraform state

# Create resource group for Terraform state
az group create --name $RESOURCE_GROUP_NAME --location eastus

Create storage account

# Create storage account
az storage account create --resource-group $RESOURCE_GROUP_NAME --name $STORAGE_ACCOUNT_NAME --sku Standard_LRS --encryption-services blob

Create Key Vault

#Create Key Vault
az keyvault create --name $KEYVAULT --resource-group $RESOURCE_GROUP_NAME --location eastus

Export Storage Account Keys

#Export Storage Account Keys
ACCOUNT_KEY=$(az storage account keys list --resource-group $RESOURCE_GROUP_NAME --account-name $STORAGE_ACCOUNT_NAME --query '[0].value' -o tsv)
export ARM_ACCESS_KEY=$ACCOUNT_KEY

Create Storage Container

#Create Storage Container
az storage container create --name $CONTAINER_NAME --account-name $STORAGE_ACCOUNT_NAME

Create Secret Key

#Create Secret Key
az keyvault secret set --name $KEYVAULTSECRET --vault-name $KEYVAULT --value tfstatesecret

Store Storage Key in a Key-Vault

#Store Storage Key in a Key-Vault
export ARM_ACCESS_KEY=$(az keyvault secret show --name $KEYVAULTSECRET --vault-name $KEYVAULT --query value -o tsv)

Verify the created keyvault resource

Verify the created resource

Terraform backend configuration in your providers.tf file

Terraform backend configuration sample – use unique names for your own configuration.

#IaC on Azure Cloud Platform | Declare Azure as the Provider
# Configure the Microsoft Azure Provider
terraform {

  required_version = ">=0.12"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~>2.0"
    }
  }

#Azurerm Backend Configuration
   backend "azurerm" {
    resource_group_name  = "remote-terraform-state"
    storage_account_name = "tfstoragetrainingenc"
    container_name       = "remote-terraform-container"
    key                  = "terraform.tfstate"
      }

}

provider "azurerm" {
  features {}
}

Create SSH Key

ssh-keygen -t rsa 4096 -f remotekey

Initialise Terraform to use the Remote State | Backend

terraform init

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the 
  newly configured "azurerm" backend. No existing state was found in the newly     
  configured "azurerm" backend. Do you want to copy this state to the new "azurerm"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes

Once the backend configuration has been placed in the provider.tf configuration, you start to see the migration away from the local state to the azurerm backend.

PS C:\Workdir\terraform\azterraform> terraform init

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "azurerm" backend. An existing non-empty state already exists in
  the new backend. The two states have been saved to temporary files that will be
  removed after responding to this query.

  Previous (type "local"): C:\Users\CLOUDA~1\AppData\Local\Temp\terraform2362619277\1-local.tfstate
  New      (type "azurerm"): C:\Users\CLOUDA~1\AppData\Local\Temp\terraform2362619277\2-azurerm.tfstate

  Do you want to overwrite the state in the new backend with the previous state?
  Enter "yes" to copy and "no" to start with the existing state in the newly
  configured "azurerm" backend.

Newly created terraform state file showing unpopulated state

Selecting yes ensures that the existing terraform state file is now copied to the backend (remote state file).

Successfully configured the backend "azurerm"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Reusing previous version of hashicorp/template from the dependency lock file
- Reusing previous version of hashicorp/azurerm from the dependency lock file
- Reusing previous version of hashicorp/random from the dependency lock file
- Reusing previous version of hashicorp/tls from the dependency lock file
- Using previously-installed hashicorp/random v3.2.0
- Using previously-installed hashicorp/tls v3.4.0
- Using previously-installed hashicorp/template v2.2.0
- Using previously-installed hashicorp/azurerm v2.99.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Pre-existing state was found while migrating the previous “local” backend to the newly configured “azurerm” backend.

Newly created terraform state file showing populated state

Verify the state of a resource – terraform state show

PS C:\Workdir\terraform\azterraform> terraform state show azurerm_resource_group.emc-eus2-corporate-import-rg
# azurerm_resource_group.emc-eus2-corporate-import-rg:
resource "azurerm_resource_group" "emc-eus2-corporate-import-rg" {
    id       = "/subscriptions/31e9c06e-6d3f-4485-836c-ff36c38135a3/resourceGroups/emc-eus2-corporate-import-rg"
    location = "eastus2"
    name     = "emc-eus2-corporate-import-rg"
    tags     = {
        "env" = "resource-group"
    }
}

You can also create your remote state with HCL as per the configuration below in a terraform file like create-remote-storage.tf.

resource "random_string" "resource_code" {
  length  = 5
  special = false
  upper   = false
}

resource "azurerm_resource_group" "tfstate" {
  name     = "tfstate"
  location = "East US"
}

resource "azurerm_storage_account" "tfstate" {
  name                     = "tfstate${random_string.resource_code.result}"
  resource_group_name      = azurerm_resource_group.tfstate.name
  location                 = azurerm_resource_group.tfstate.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
  allow_blob_public_access = true

  tags = {
    environment = "staging"
  }
}

resource "azurerm_storage_container" "tfstate" {
  name                  = "tfstate"
  storage_account_name  = azurerm_storage_account.tfstate.name
  container_access_type = "blob"
}

Below are useful Terraform state commands

terraform state pull

This command downloads the state from its current location, upgrades the local copy to the latest state file version that is compatible with locally-installed Terraform, and outputs the raw format to stdout.

terraform state pull | sc terraform.tfstate

Troubleshooting Remote State Locks

╷
│ Error: Error acquiring the state lock
│
│ Error message: state blob is already locked
│ Lock Info:
│   ID:        42ff26c4-3103-e54c-e21a-8a37a25e5884
│   Path:      tfbackendcontainer/terraform.tfstate
│   Operation: OperationTypeApply
│   Who:       DESKTOP-ATCRUJV\Cloud Architect@DESKTOP-ATCRUJV
│   Version:   1.1.9
│   Created:   2022-06-06 22:53:18.7745943 +0000 UTC
│   Info:
│
│
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.

terraform destroy -lock=false

│ Error: Failed to save state
│
│ Error saving state: blobs.Client#PutBlockBlob: Failure responding to request: StatusCode=412 -- Original Error: autorest/azure: Service returned an error. Status=412 Code="LeaseIdMissing" Message="There is currently a    
│ lease on the blob and no lease ID was specified in the request.\nRequestId:110febce-c01e-000b-1326-7a1b95000000\nTime:2022-06-07T04:27:53.9123101Z"
╵

╷
│ Error: Failed to persist state to backend
│
│ The error shown above has prevented Terraform from writing the updated state to the configured backend. To allow for recovery, the state has been written to the file "errored.tfstate" in the current working directory.    
│
│ Running "terraform apply" again at this point will create a forked state, making it harder to recover.
│
│ To retry writing this state, use the following command:
│     terraform state push errored.tfstate

The lock has occurred because there was a terraform action initiated by a user which has placed that lock. To fix it, go to Azure and on the blob break the lease and delete the lock.