Context

• Managing thousands of clusters at STACKIT Kubernetes Engine
• Based on open source project Gardener
• Running controllers at scale
• Master’s thesis: “Horizontally Scalable Kubernetes Controllers”

Controller Basics

Controllers facilitate declarative state management:

1. Watch objects for changes
2. Cache objects in memory
3. Enqueue object key on relevant changes
4. Read current state (from cache)
5. Make changes
6. Report observed status

Scalability Challenges

• Controllers must prevent conflicts
• Perform leader election
• Only a single active instance
• Controllers are not horizontally scalable
• Limits large-scale use cases
• No standard solution exists

Key Features

• Sharding mechanisms inspired by distributed databases
• Dynamic membership and failure detection
• Automatic failover and rebalancing
• Transparent label-based coordination
• Prevents concurrent reconciliations
• Reusable implementation

Architecture

1) Install Sharding Components

kubectl apply --server-side –k \
  "https://github.com/timebertt/kubernetes-controller-sharding//config/default"

2) ControllerRing

apiVersion: sharding.timebertt.dev/v1alpha1
kind: ControllerRing
metadata:
  name: webhosting-operator
spec:
  resources:
  - group: webhosting.timebertt.dev
    resource: websites
    controlledResources:
    - group: apps
      resource: deployments
    - group: networking.k8s.io
      resource: ingresses
    # ...
apiVersion: sharding.timebertt.dev/v1alpha1
kind: ControllerRing
metadata:
  name: webhosting-operator
spec:
  resources:
  - group: webhosting.timebertt.dev
    resource: websites
    controlledResources:
    - group: apps
      resource: deployments
    - group: networking.k8s.io
      resource: ingresses
    # ...
apiVersion: sharding.timebertt.dev/v1alpha1
kind: ControllerRing
metadata:
  name: webhosting-operator
spec:
  resources:
  - group: webhosting.timebertt.dev
    resource: websites
    controlledResources:
    - group: apps
      resource: deployments
    - group: networking.k8s.io
      resource: ingresses
    # ...

3) Shard Lease

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  labels:
    alpha.sharding.timebertt.dev/controllerring: webhosting-operator
  name: webhosting-operator-65ffcdb674-8sst9
  namespace: webhosting-system
spec:
  holderIdentity: webhosting-operator-65ffcdb674-8sst9
  acquireTime: "2025-04-03T10:45:51.992779Z"
  renewTime: "2025-04-03T11:02:26.817751Z"
  leaseDurationSeconds: 15
apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  labels:
    alpha.sharding.timebertt.dev/controllerring: webhosting-operator
  name: webhosting-operator-65ffcdb674-8sst9
  namespace: webhosting-system
spec:
  holderIdentity: webhosting-operator-65ffcdb674-8sst9
  acquireTime: "2025-04-03T10:45:51.992779Z"
  renewTime: "2025-04-03T11:02:26.817751Z"
  leaseDurationSeconds: 15
apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  labels:
    alpha.sharding.timebertt.dev/controllerring: webhosting-operator
  name: webhosting-operator-65ffcdb674-8sst9
  namespace: webhosting-system
spec:
  holderIdentity: webhosting-operator-65ffcdb674-8sst9
  acquireTime: "2025-04-03T10:45:51.992779Z"
  renewTime: "2025-04-03T11:02:26.817751Z"
  leaseDurationSeconds: 15

3) Shard Lease (Go)

shardLease, err := shardlease.NewResourceLock(restConfig, shardlease.Options{
  ControllerRingName: "webhosting-operator",
})
if err != nil {
  return err
}
 
mgr, err := manager.New(restConfig, manager.Options{
  LeaderElection: true,
  LeaderElectionResourceLockInterface: shardLease,
  LeaderElectionReleaseOnCancel: true,
  // ...
})
if err != nil {
  return err
}
shardLease, err := shardlease.NewResourceLock(restConfig, shardlease.Options{
  ControllerRingName: "webhosting-operator",
})
if err != nil {
  return err
}
 
mgr, err := manager.New(restConfig, manager.Options{
  LeaderElection: true,
  LeaderElectionResourceLockInterface: shardLease,
  LeaderElectionReleaseOnCancel: true,
  // ...
})
if err != nil {
  return err
}
shardLease, err := shardlease.NewResourceLock(restConfig, shardlease.Options{
  ControllerRingName: "webhosting-operator",
})
if err != nil {
  return err
}
 
mgr, err := manager.New(restConfig, manager.Options{
  LeaderElection: true,
  LeaderElectionResourceLockInterface: shardLease,
  LeaderElectionReleaseOnCancel: true,
  // ...
})
if err != nil {
  return err
}

4) Filter Watch Cache

Only watch and reconcile objects with this label:

shard.alpha.sharding.timebertt.dev/webhosting-operator: "<shard-name>"

4) Filter Watch Cache (Go)

labelSelector := labels.SelectorFromSet(labels.Set{
  // shard.alpha.sharding.timebertt.dev/webhosting-operator=<shard-name>
  shardingv1alpha1.LabelShard("webhosting-operator"): shardLease.Identity(),
})
 
mgr, err := manager.New(restConfig, manager.Options{
  Cache: cache.Options{
    DefaultLabelSelector: labelSelector,
  },
})
if err != nil {
  return err
}
labelSelector := labels.SelectorFromSet(labels.Set{
  // shard.alpha.sharding.timebertt.dev/webhosting-operator=<shard-name>
  shardingv1alpha1.LabelShard("webhosting-operator"): shardLease.Identity(),
})
 
mgr, err := manager.New(restConfig, manager.Options{
  Cache: cache.Options{
    DefaultLabelSelector: labelSelector,
  },
})
if err != nil {
  return err
}

5) Drain Operations

Stop reconciling objects with this label and remove the labels:

drain.alpha.sharding.timebertt.dev/webhosting-operator: "true"

5) Drain Operations (Go)

builder.ControllerManagedBy(mgr).
  For(&webhostingv1alpha1.Website{}, builder.WithPredicates(
    WebsitePredicate(),
  )).
  Owns(&appsv1.Deployment{}, builder.WithPredicates(DeploymentPredicate())).
  Complete(
    reconciler,
  )
builder.ControllerManagedBy(mgr).
  For(&webhostingv1alpha1.Website{}, builder.WithPredicates(
    WebsitePredicate(),
  )).
  Owns(&appsv1.Deployment{}, builder.WithPredicates(DeploymentPredicate())).
  Complete(
    reconciler,
  )

5) Drain Operations (Go)

builder.ControllerManagedBy(mgr).
  For(&webhostingv1alpha1.Website{}, builder.WithPredicates(
    shardcontroller.Predicate(controllerRing, shardName, WebsitePredicate()),
  )).
  Owns(&appsv1.Deployment{}, builder.WithPredicates(DeploymentPredicate())).
  Complete(
    reconciler,
  )
builder.ControllerManagedBy(mgr).
  For(&webhostingv1alpha1.Website{}, builder.WithPredicates(
    shardcontroller.Predicate(controllerRing, shardName, WebsitePredicate()),
  )).
  Owns(&appsv1.Deployment{}, builder.WithPredicates(DeploymentPredicate())).
  Complete(
    reconciler,
  )

5) Drain Operations (Go)

builder.ControllerManagedBy(mgr).
  For(&webhostingv1alpha1.Website{}, builder.WithPredicates(
    shardcontroller.Predicate(controllerRing, shardName, WebsitePredicate()),
  )).
  Owns(&appsv1.Deployment{}, builder.WithPredicates(DeploymentPredicate())).
  Complete(
    shardcontroller.NewShardedReconciler(mgr).
      For(&webhostingv1alpha1.Website{}).
      InControllerRing(controllerRing).
      WithShardName(shardName).
      MustBuild(reconciler),
  )

Load Tests

• Dimension 1: object count – up to 9,000 objects
• Dimension 2: object churn – up to 300 changes per second

Resource Usage

Performance Measurements

queries:
# SLO 1: Queue Latency: p99 < 1s
- name: latency-queue
  query: |
    histogram_quantile(0.99, sum by (le) (
      workqueue_queue_duration_seconds_bucket{
        job="webhosting-operator", name="website"
      }
    ))
 # SLO 2: Reconciliation Latency: p99 < 5s
- name: latency-reconciliation
  query: |
    histogram_quantile(0.99, sum by (le) (
      experiment_website_reconciliation_duration_seconds_bucket{
        job="experiment"
      }
    ))
queries:
# SLO 1: Queue Latency: p99 < 1s
- name: latency-queue
  query: |
    histogram_quantile(0.99, sum by (le) (
      workqueue_queue_duration_seconds_bucket{
        job="webhosting-operator", name="website"
      }
    ))
 # SLO 2: Reconciliation Latency: p99 < 5s
- name: latency-reconciliation
  query: |
    histogram_quantile(0.99, sum by (le) (
      experiment_website_reconciliation_duration_seconds_bucket{
        job="experiment"
      }
    ))
queries:
# SLO 1: Queue Latency: p99 < 1s
- name: latency-queue
  query: |
    histogram_quantile(0.99, sum by (le) (
      workqueue_queue_duration_seconds_bucket{
        job="webhosting-operator", name="website"
      }
    ))
 # SLO 2: Reconciliation Latency: p99 < 5s
- name: latency-reconciliation
  query: |
    histogram_quantile(0.99, sum by (le) (
      experiment_website_reconciliation_duration_seconds_bucket{
        job="experiment"
      }
    ))

Load Capacity

Webhosting-Operator

Use CreateOrPatch instead of no-op APPLY calls

Performance Improvements

Remove client-side rate limiting (#610), Improve sharder retries (#636), Move and drain objects concurrently (#637)

Performance Improvements

Chaos Experiments

Kubebuilder Plugin

Partitioning by Owner of Owner

Stabilize Rolling Updates

Horizontal Autoscaling

Conclusion

• Design makes Kubernetes controllers horizontally scalable
• Capacity increases almost linearly with every added instance
• Applicable to arbitrary controllers
• Simple implementation
• Ready for usage and collaboration in the open source community
• Gather experience in real-world scenarios

Leave a star!

Tim Ebert, STACKIT

timebertt/kubernetes-controller-sharding

@timebertt

@timebertt