Kubernetes as Workload on Kubernetes

Rotation At Scale

• Gardener manages thousands of clusters

• How to orchestrate credentials rotation at such scale?

• Goals: Disruption-free, minimal ops, fully automated

Projected Token Mount 🔮

Since Kubernetes v1.22, this results in

spec:
  serviceAccountName: robot
  containers:
  - name: nginx
    image: nginx
    volumeMounts:
    - name: kube-api-access-5f65c
      mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      readOnly: true
  volumes:
  - name: kube-api-access-5f65c
    projected:
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
spec:
  serviceAccountName: robot
  containers:
  - name: nginx
    image: nginx
    volumeMounts:
    - name: kube-api-access-5f65c
      mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      readOnly: true
  volumes:
  - name: kube-api-access-5f65c
    projected:
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
spec:
  serviceAccountName: robot
  containers:
  - name: nginx
    image: nginx
    volumeMounts:
    - name: kube-api-access-5f65c
      mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      readOnly: true
  volumes:
  - name: kube-api-access-5f65c
    projected:
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace

TokenRequest API

Projected Token Mount 🔮

Since Kubernetes v1.22, this results in

spec:
  containers:
  - name: nginx
    image: nginx
    volumeMounts:
    - name: kube-api-access-5f65c
      mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      readOnly: true
  volumes:
  - name: kube-api-access-5f65c
    projected:
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
spec:
  containers:
  - name: nginx
    image: nginx
    volumeMounts:
    - name: kube-api-access-5f65c
      mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      readOnly: true
  volumes:
  - name: kube-api-access-5f65c
    projected:
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace

"Magic" Expiration Time 🪄

const (
    WarnOnlyBoundTokenExpirationSeconds = 60*60+7 // 3607
    ExpirationExtensionSeconds          = 24*365*60*60 // 1y
)
 
exp := req.Spec.ExpirationSeconds
 
if pod != nil &&
  r.isKubeAudiences(req.Spec.Audiences) &&
  r.extendExpiration &&
  req.Spec.ExpirationSeconds == WarnOnlyBoundTokenExpirationSeconds {
 
  exp = ExpirationExtensionSeconds
}
const (
    WarnOnlyBoundTokenExpirationSeconds = 60*60+7 // 3607
    ExpirationExtensionSeconds          = 24*365*60*60 // 1y
)
 
exp := req.Spec.ExpirationSeconds
 
if pod != nil &&
  r.isKubeAudiences(req.Spec.Audiences) &&
  r.extendExpiration &&
  req.Spec.ExpirationSeconds == WarnOnlyBoundTokenExpirationSeconds {
 
  exp = ExpirationExtensionSeconds
}
const (
    WarnOnlyBoundTokenExpirationSeconds = 60*60+7 // 3607
    ExpirationExtensionSeconds          = 24*365*60*60 // 1y
)
 
exp := req.Spec.ExpirationSeconds
 
if pod != nil &&
  r.isKubeAudiences(req.Spec.Audiences) &&
  r.extendExpiration &&
  req.Spec.ExpirationSeconds == WarnOnlyBoundTokenExpirationSeconds {
 
  exp = ExpirationExtensionSeconds
}
const (
    WarnOnlyBoundTokenExpirationSeconds = 60*60+7 // 3607
    ExpirationExtensionSeconds          = 24*365*60*60 // 1y
)
 
exp := req.Spec.ExpirationSeconds
 
if pod != nil &&
  r.isKubeAudiences(req.Spec.Audiences) &&
  r.extendExpiration &&
  req.Spec.ExpirationSeconds == WarnOnlyBoundTokenExpirationSeconds {
 
  exp = ExpirationExtensionSeconds
}

kube-apiserver silently overwrites the expiration seconds 👻 Source

Conclusion

• Set --service-account-extend-token-expiration=false to ensure tokens are indeed only valid for 1h

• If you cannot control the flag (e.g., in managed clusters), overwrite the expirationSeconds to ensure short validity

KEP-2799: Reduction of Secret-based Service Account Tokens

Most probably static tokens still exist in your clusters! 👹

Conclusion

• If you are on v1.24 or higher, manually delete still remaining static token secrets

• If you are stuck below v1.24, consider invalidating the tokens (talk to us!)

Solution 💡

Rotation in two phases:

1. issue new credentials, accept both old and new
2. invalidate old credentials

Server Certificates 🗄

Client Certificates 🧑‍💻

Key Elements 🔑

• clients need to refresh their credentials after preparation
• clients trigger completion once ready
• automatic rotation for non-user-facing credentials

Requesting a Server Cert

Live Coding!

Next Level: Auto-Rotation 🔁

• activated for non-user-facing credentials
• phase 1 triggered when approaching end of validity
• phase 2 triggered 24h later
• fully-automated + disruption-free!

Key Takeaways

• use short-lived projected ServiceAccount tokens
• delete remaining static ServiceAccount tokens

Key Takeaways

• rotate static credentials in two phases
• use bundles in between
• implemented for Kubernetes as workload, concept is applicable to most workloads