Skip to main content

Temporal Worker Controller

Temporal's deterministic constraints require consideration when rolling out or rolling back workflow code changes.

The traditional approach to workflow determinism is to add branches using the Versioning APIs. Over time these checks can become a source of technical debt, as safely removing them from a codebase is a careful process that often involves querying all running workflows.

Worker Versioning is a Temporal feature that allows you to pin Workflows to individual versions of your workers, which are called Worker Deployment Versions. Using pinning, you will not need to add branching to your Workflows to avoid non-determinism errors.

This allows you to bypass the other Versioning APIs. Instead, your deployment system must support multiple versions running simultaneously and allow you to control when they are sunsetted.

This Temporal Worker Controller provides automation to enable rainbow deployments of your workers by simplifying the tracking of which versions still have active workflows, managing the lifecycle of versioned worker deployments, and calling Temporal APIs to update the routing config of Temporal Worker Deployments. This way, you can route Workflow traffic to new versions.

Note that in Temporal, Worker Deployment is sometimes referred to as Deployment, but since the controller makes significant references to Kubernetes Deployment resource, within this page we will stick to these terms:

  • Worker Deployment Version: A version of a deployment or service. It can have multiple Workers, but they all run the same build. Sometimes shortened to "version" or "deployment version."
  • Worker Deployment: A deployment or service across multiple versions. In a rainbow deploy, a worker deployment can have multiple active deployment versions running at once.
  • Deployment: A Kubernetes Deployment resource. A Deployment is "versioned" if it is running versioned Temporal workers/pollers.

Features​

  • Registration of new Temporal Worker Deployment Versions
  • Creation of versioned Deployment resources (that manage the Pods that run your Temporal pollers)
  • Deletion of resources associated with drained Worker Deployment Versions
  • Manual, AllAtOnce, and Progressive rollouts of new versions
  • Ability to specify a "gate" workflow that must succeed on the new version before routing real traffic to that version
  • (Coming soon) Autoscaling of versioned Deployments

Configuring Worker Lifecycle​

To use the Temporal Worker Controller, tag your Workers following the guidance for using Worker Versioning. Workers need to be configured using these environment variables:

  • TEMPORAL_HOST_PORT: The host and port of the Temporal server, e.g. default.foo.tmprl.cloud:7233
  • TEMPORAL_NAMESPACE: The Temporal namespace to connect to, e.g. default
  • TEMPORAL_DEPLOYMENT_NAME: The name of the worker deployment. This must be unique to the worker deployment and should not change between versions.
  • WORKER_BUILD_ID: The build ID of the worker. This should change with each new worker rollout. Each of these will be automatically set by the controller, and must not be manually specified in the worker's pod template.

When a new worker deployment version is deployed, the worker controller detects it and automatically begins the process of making that version the new Current Version of the worker deployment it is a part of. This could happen immediately if cutover.strategy = AllAtOnce, or gradually if cutover.strategy = Progressive.

As older pinned workflows finish executing and deprecated deployment versions become drained, the worker controller also frees up resources by sunsetting the Deployment resources polling those versions.

Here is an example of a progressive cut-over strategy gated on the success of the HelloWorld workflow:

cutover:
  strategy: Progressive
  steps:
    - rampPercentage: 1
      pauseDuration: 30s
    - rampPercentage: 10
      pauseDuration: 1m
  gate:
    workflowType: "HelloWorld"

Running the Temporal Worker Controller​

You can install the Temporal Worker Controller using our helm chart:

RELEASE=temporal-worker-controller
NAMESPACE=temporal-system
VERSION=1.0.0

helm install $RELEASE oci://docker.io/temporalio/helm-charts/temporal-worker-controller \
  --version $VERSION \
  --namespace $NAMESPACE \
  --create-namespace
  
helm install temporal-worker-controller ./helm/temporal-worker-controller \
  --namespace temporal-system \
  --create-namespace

Refer to Github for other Worker Controller deployment templates.