When you hear secrets, what comes to mind is confidentiality and secrecy. In the world of Kubernetes secrets are essentially any value that you don’t want the world to know about.

The following elements, password, an API key, a connection string to a database, all fall under what a secret is. Now when comparing Secrets and ConfigMaps in Kubernetes, the main difference is the confidential data.

Both ConfigMaps and Secrets store the data the same way, with key/value pairs, but ConfigMaps are designed for plain text data, and secrets on the other hand are meant for data that must be secured and confidential to the application exclusively.

By default, Secrets are stored at rest in Key Vault, in a secure encrypted store. Secrets are only stored in the AKS cluster when a pod is running with the secret mounted as a volume in a pod. As soon as the hosting pods are removed, the secret is removed from the cluster and this is a better approach as opposed to Kubernetes secrets which gets retained after the hosting pod is removed.

RESOURCE_GROUP=corp-infrastructure-rg
KV_RESOURCE_GROUP=corp-kv-infrastructure-rg
LOCATION=eastus
AKS_CLUSTER=corpakscluster

#Create a resource group for the AKS cluster:

az group create --name $RESOURCE_GROUP --location $LOCATION

 az aks create \
   --resource-group $RESOURCE_GROUP \
   --name $AKS_CLUSTER \
   --network-plugin azure \
   --enable-managed-identity \
   --enable-addons azure-keyvault-secrets-provider \
   --generate-ssh-keys

 "identity": {
        "clientId": "1456c162-3f04-40bc-a079-f1f3f7d22b16",
        "objectId": "9f8165b6-206f-4596-932f-31e80469700f",
}

Download the cluster credentials and configure kubectl to use them:

az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER

Merged "corpakscluster" as current context in /home/%user%/.kube/config

Check that the Secrets Store CSI Driver and the Azure Key Vault Provider are installed in the cluster:

$ kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver, secrets-store-provider-azure)'

When we enable the Azure Key Vault secret provider, the add-on will create a user assigned managed identity in the node managed resource group. Store its resource ID in a variable for later use

View the resource ID of the user assigned managed identity;

az aks show -g $RESOURCE_GROUP -n $AKS_CLUSTER --query addonProfiles.azureKeyvaultSecretsProvider.identity.clientId -o tsv
1456c162-3f04-40bc-a079-f1f3f7d22b16

Store the resource ID of the user assigned managed identity in a variable;

KV_IDENTITY_RESOURCE_ID=$(az aks show -g $RESOURCE_GROUP -n $AKS_CLUSTER --query addonProfiles.azureKeyvaultSecretsProvider.identity.clientId -o tsv)

Create Azure Key Vault
Create a resource group for Azure Key vault

az group create --name $KV_RESOURCE_GROUP --location $LOCATION

Create a key vault while storing its name in a variable:

KEY_VAULT_NAME="akscorpkeyvault${RANDOM}"

az keyvault create --name $KEY_VAULT_NAME --resource-group $KV_RESOURCE_GROUP --location $LOCATION

{
 "name": "akscorpkeyvault5493"
"objectId": "ebejced9-2f89-8176-a9u3-657f75eb36bb"
"tenantId": "46edb775-xy69-41z6-7be1-03e4a0997e49"
}

Create a secret and a key in the Vault for later demonstration:

az keyvault secret set --vault-name $KEY_VAULT_NAME -n FirstSecret --value StoredValueinFirstSecret

 "name": "FirstSecret",
  "tags": {
    "file-encoding": "utf-8"
  },
  "value": "StoredValueinFirstSecret"
}

Create a key in the Vault for later demonstration:

az keyvault key create --vault-name $KEY_VAULT_NAME -n FirstKey --protection software

    "n": "t6PMnN5hTR2Oicy/fuTzQgXo49EgkS7B61gJWOeQjfw8u9tO+YoRbnPgWMnDsQWE3xE/MJyt6R0w0QwHsQa28KjdzCfq6qvJSlTSyhFfU9VJIf2YkjFtSlOpoyqYXKmHC6cS3pLrWsxDdVZTpZrgcZ8ec2deowrLDnn9mL5OKljGHmEaptocVHGWGfs9VNlxNqDAhRC4IKCQSIt6pnXc+eLo6Es0J50WhqHTGdqMG5brJGSlgEVaZobeBuvyFIxEvtt33MDjjkdiXCjKoTl8IS7/LNlvLYtDTWRvazK390IUXpldICw0xAp3layR/IDZA0diLEwQzbdESkyO18osPQ==",

Grant the AKS key vault managed identity permissions to read (GET) your key vault and view its contents:

Set policy to access keys in your key vault

az keyvault set-policy -n $KEY_VAULT_NAME --key-permissions get --spn $KV_IDENTITY_RESOURCE_ID

"objectId": "ebejced9-2f89-8176-a9u3-657f75eb36bb", granted the permissions to read the object     "objectId": "9f8165b6-206f-4596-932f-31e80469700f"

 "keys": [
            "get"
          ],

Set policy to access secrets in your key vault

az keyvault set-policy -n $KEY_VAULT_NAME --secret-permissions get --spn $KV_IDENTITY_RESOURCE_ID

"objectId": "ebejced9-2f89-8176-a9u3-657f75eb36bb", granted the permissions to read the object     "objectId": "9f8165b6-206f-4596-932f-31e80469700f"

"secrets": [
            "get"
          ]

Set policy to access certs in your key vault

az keyvault set-policy -n $KEY_VAULT_NAME --certificate-permissions get --spn $KV_IDENTITY_RESOURCE_ID

 "certificates": [
            "get"
          ],

Create Kubernetes resources
Store the tenant ID in a variable, you can get the value from the Azure AD tenant overview page:
TENANT_ID=${{put your tenant ID here}}  | TENANT_ID=46edb775-xy69-41z6-7be1-03e4a0997e49

Create a SecretProviderClass by using the following YAML, using your own values for userAssignedIdentityID, keyvaultName, tenantId, and the objects to retrieve from your key vault:

cat <<EOF | kubectl apply -f -
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-user-msi
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true" # true since using managed identity
    userAssignedIdentityID: 1456c162-3f04-40bc-a079-f1f3f7d22b16 #$KV_IDENTITY_RESOURCE_ID
    keyvaultName: akscorpkeyvault5493    #$KEY_VAULT_NAME
    cloudName: ""
    objects:  |
      array:
        - |
          objectName: FirstSecret        #ExampleSecret
          objectType: secret    # object types: secret, key, or cert
          objectVersion: ""     # default to latest if empty
        - |
          objectName: FirstKey        #ExampleKey
          objectType: key
          objectVersion: ""
    tenantId: 46edb775-xy69-41z6-7be1-03e4a0997e49 #$TENANT_ID
EOF

secretproviderclass.secrets-store.csi.x-k8s.io/azure-kvname-user-msi configured

At this point, you need a pod that mounts the secret and the key using the secret provider class we just created earlier above:

cat <<EOF | kubectl apply -f -
---
kind: Pod
apiVersion: v1
metadata:
  name: busybox-secrets-store-inline-user-msi
spec:
  containers:
    - name: busybox
      image: k8s.gcr.io/e2e-test-images/busybox:1.29-1
      command:
        - "/bin/sleep"
        - "10000"
      volumeMounts:
      - name: secrets-store01-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-kvname-user-msi"
EOF


pod/busybox-secrets-store-inline-user-msi created

Validate secrets were mounted from the pod created earlier:

kubectl exec busybox-secrets-store-inline-user-msi -- ls /mnt/secrets-store/

Read the content(s) of the secret and key:

kubectl exec busybox-secrets-store-inline-user-msi -- cat /mnt/secrets-store/FirstSecret

kubectl exec busybox-secrets-store-inline-user-msi -- cat /mnt/secrets-store/FirstKey