Before we begin to discuss how to access secrets from Azure Key Vault in Azure Kubernetes Service, let us have a quick intro to Secrets in Kubernetes.
When you hear secrets, what comes to mind is confidentiality and secrecy. In the world of Kubernetes secrets are essentially any value that you don’t want the world to know about.
The following elements, password, an API key, a connection string to a database, all fall under what a secret is. Now when comparing Secrets and ConfigMaps in Kubernetes, the main difference is the confidential data.
Both ConfigMaps and Secrets store the data the same way, with key/value pairs, but ConfigMaps are designed for plain text data, and secrets on the other hand are meant for data that must be secured and confidential to the application exclusively.
By default, Secrets are stored at rest in Key Vault, in a secure encrypted store. Secrets are only stored in the AKS cluster when a pod is running with the secret mounted as a volume in a pod. As soon as the hosting pods are removed, the secret is removed from the cluster and this is a better approach as opposed to Kubernetes secrets which gets retained after the hosting pod is removed.
RESOURCE_GROUP=corp-infrastructure-rg KV_RESOURCE_GROUP=corp-kv-infrastructure-rg LOCATION=eastus AKS_CLUSTER=corpakscluster
#Create a resource group for the AKS cluster:
az group create --name $RESOURCE_GROUP --location $LOCATION
az aks create \ --resource-group $RESOURCE_GROUP \ --name $AKS_CLUSTER \ --network-plugin azure \ --enable-managed-identity \ --enable-addons azure-keyvault-secrets-provider \ --generate-ssh-keys
"identity": {
"clientId": "1456c162-3f04-40bc-a079-f1f3f7d22b16",
"objectId": "9f8165b6-206f-4596-932f-31e80469700f",
}
Download the cluster credentials and configure kubectl to use them:
az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER
Merged "corpakscluster" as current context in /home/%user%/.kube/config
Check that the Secrets Store CSI Driver and the Azure Key Vault Provider are installed in the cluster:
$ kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver, secrets-store-provider-azure)'
When we enable the Azure Key Vault secret provider, the add-on will create a user assigned managed identity in the node managed resource group. Store its resource ID in a variable for later use
View the resource ID of the user assigned managed identity;
az aks show -g $RESOURCE_GROUP -n $AKS_CLUSTER --query addonProfiles.azureKeyvaultSecretsProvider.identity.clientId -o tsv 1456c162-3f04-40bc-a079-f1f3f7d22b16
Store the resource ID of the user assigned managed identity in a variable;
KV_IDENTITY_RESOURCE_ID=$(az aks show -g $RESOURCE_GROUP -n $AKS_CLUSTER --query addonProfiles.azureKeyvaultSecretsProvider.identity.clientId -o tsv)
Create Azure Key Vault
Create a resource group for Azure Key vault
az group create --name $KV_RESOURCE_GROUP --location $LOCATION
Create a key vault while storing its name in a variable:
KEY_VAULT_NAME="akscorpkeyvault${RANDOM}"
az keyvault create --name $KEY_VAULT_NAME --resource-group $KV_RESOURCE_GROUP --location $LOCATION
{
"name": "akscorpkeyvault5493"
"objectId": "ebejced9-2f89-8176-a9u3-657f75eb36bb"
"tenantId": "46edb775-xy69-41z6-7be1-03e4a0997e49"
}
Create a secret and a key in the Vault for later demonstration:
az keyvault secret set --vault-name $KEY_VAULT_NAME -n FirstSecret --value StoredValueinFirstSecret
"name": "FirstSecret",
"tags": {
"file-encoding": "utf-8"
},
"value": "StoredValueinFirstSecret"
}
Create a key in the Vault for later demonstration:
az keyvault key create --vault-name $KEY_VAULT_NAME -n FirstKey --protection software
"n": "t6PMnN5hTR2Oicy/fuTzQgXo49EgkS7B61gJWOeQjfw8u9tO+YoRbnPgWMnDsQWE3xE/MJyt6R0w0QwHsQa28KjdzCfq6qvJSlTSyhFfU9VJIf2YkjFtSlOpoyqYXKmHC6cS3pLrWsxDdVZTpZrgcZ8ec2deowrLDnn9mL5OKljGHmEaptocVHGWGfs9VNlxNqDAhRC4IKCQSIt6pnXc+eLo6Es0J50WhqHTGdqMG5brJGSlgEVaZobeBuvyFIxEvtt33MDjjkdiXCjKoTl8IS7/LNlvLYtDTWRvazK390IUXpldICw0xAp3layR/IDZA0diLEwQzbdESkyO18osPQ==",
Grant the AKS key vault managed identity permissions to read (GET) your key vault and view its contents:
Set policy to access keys in your key vault
az keyvault set-policy -n $KEY_VAULT_NAME --key-permissions get --spn $KV_IDENTITY_RESOURCE_ID
"objectId": "ebejced9-2f89-8176-a9u3-657f75eb36bb", granted the permissions to read the object "objectId": "9f8165b6-206f-4596-932f-31e80469700f"
"keys": [
"get"
],
Set policy to access secrets in your key vault
az keyvault set-policy -n $KEY_VAULT_NAME --secret-permissions get --spn $KV_IDENTITY_RESOURCE_ID
"objectId": "ebejced9-2f89-8176-a9u3-657f75eb36bb", granted the permissions to read the object "objectId": "9f8165b6-206f-4596-932f-31e80469700f"
"secrets": [
"get"
]
Set policy to access certs in your key vault
az keyvault set-policy -n $KEY_VAULT_NAME --certificate-permissions get --spn $KV_IDENTITY_RESOURCE_ID
"certificates": [
"get"
],
Create Kubernetes resources
Store the tenant ID in a variable, you can get the value from the Azure AD tenant overview page:
TENANT_ID=${{put your tenant ID here}} | TENANT_ID=46edb775-xy69-41z6-7be1-03e4a0997e49
Create a SecretProviderClass by using the following YAML, using your own values for userAssignedIdentityID, keyvaultName, tenantId, and the objects to retrieve from your key vault:
cat <<EOF | kubectl apply -f -
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: azure-kvname-user-msi
spec:
provider: azure
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true" # true since using managed identity
userAssignedIdentityID: 1456c162-3f04-40bc-a079-f1f3f7d22b16 #$KV_IDENTITY_RESOURCE_ID
keyvaultName: akscorpkeyvault5493 #$KEY_VAULT_NAME
cloudName: ""
objects: |
array:
- |
objectName: FirstSecret #ExampleSecret
objectType: secret # object types: secret, key, or cert
objectVersion: "" # default to latest if empty
- |
objectName: FirstKey #ExampleKey
objectType: key
objectVersion: ""
tenantId: 46edb775-xy69-41z6-7be1-03e4a0997e49 #$TENANT_ID
EOF
secretproviderclass.secrets-store.csi.x-k8s.io/azure-kvname-user-msi configured
At this point, you need a pod that mounts the secret and the key using the secret provider class we just created earlier above:
cat <<EOF | kubectl apply -f -
---
kind: Pod
apiVersion: v1
metadata:
name: busybox-secrets-store-inline-user-msi
spec:
containers:
- name: busybox
image: k8s.gcr.io/e2e-test-images/busybox:1.29-1
command:
- "/bin/sleep"
- "10000"
volumeMounts:
- name: secrets-store01-inline
mountPath: "/mnt/secrets-store"
readOnly: true
volumes:
- name: secrets-store01-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "azure-kvname-user-msi"
EOF
pod/busybox-secrets-store-inline-user-msi created
Validate secrets were mounted from the pod created earlier:
kubectl exec busybox-secrets-store-inline-user-msi -- ls /mnt/secrets-store/
Read the content(s) of the secret and key:
kubectl exec busybox-secrets-store-inline-user-msi -- cat /mnt/secrets-store/FirstSecret
kubectl exec busybox-secrets-store-inline-user-msi -- cat /mnt/secrets-store/FirstKey