Workers

Workers are the processes that actually execute your tasks. Each worker is a long-running process in your infrastructure that maintains a persistent gRPC connection to the Hatchet engine. Workers receive task assignments, run your code, and report results back. You can run them locally during development, in containers, or on VMs - and scale them independently from the rest of your stack.

Declaring a worker

A worker needs a name and a set of tasks to handle. Call the worker method on the Hatchet client with both.

def main() -> None:
    worker = hatchet.worker("dag-worker", workflows=[dag_workflow])

    worker.start()

import { hatchet } from '../hatchet-client';
import { simple } from './workflow';
import { parent, child } from './workflow-with-child';
import { simpleWithZod } from './zod';

async function main() {
  const worker = await hatchet.worker('simple-worker', {
    // 👀 Declare the workflows that the worker can execute
    workflows: [simple, simpleWithZod, parent, child],
    // 👀 Declare the number of concurrent task runs the worker can accept
    slots: 100,
  });

  await worker.start();
}

if (require.main === module) {
  main();
}

worker, err := client.NewWorker("simple-worker", hatchet.WithWorkflows(task))
if err != nil {
	log.Fatalf("failed to create worker: %v", err)
}

interruptCtx, cancel := cmdutils.NewInterruptContext()
defer cancel()

err = worker.StartBlocking(interruptCtx)
if err != nil {
	log.Fatalf("failed to start worker: %v", err)
}

def main
  worker = HATCHET.worker("dag-worker", workflows: [DAG_WORKFLOW])
  worker.start
end

When a worker starts, it registers each of its tasks with the Hatchet engine. From that point on, Hatchet knows to route matching tasks to that worker. Multiple workers can register the same task - Hatchet distributes work across all of them.

Starting a worker

hatchet worker dev

export HATCHET_CLIENT_TOKEN="<your-client-token>"

export HATCHET_CLIENT_TLS_STRATEGY=none

python worker.py

Once the worker starts, you will see logs confirming it is connected:

[INFO]  🪓 -- STARTING HATCHET...
[DEBUG] 🪓 -- 'test-worker' waiting for ['simpletask:step1']
[DEBUG] 🪓 -- acquired action listener: efc4aaf2-...
[DEBUG] 🪓 -- sending heartbeat

Worker lifecycle

A worker moves through four phases during its lifetime:

• ACTIVE - the worker is connected and accepting tasks.
• INACTIVE - the engine has not received a heartbeat within the expected window. Tasks assigned to this worker will be reassigned.
• STOPPED - the worker shut down gracefully. In-flight tasks are allowed to complete before the process exits.

Hatchet uses heartbeats to monitor worker health. Workers send a heartbeat every 4 seconds. If the engine does not receive a heartbeat for 30 seconds, the worker is marked INACTIVE and its in-flight tasks are re-queued for other workers to pick up.

Common reasons a worker misses heartbeats:

• Process crash - the worker process exits unexpectedly (OOM kill, unhandled exception, SIGKILL).
• Network disruption - the connection between the worker and the Hatchet engine is interrupted (DNS failure, firewall change, cloud network blip).
• Blocked main thread - a long-running synchronous computation (e.g. CPU-intensive work, a blocking FFI call) starves the heartbeat loop and prevents it from sending on time.

Slots

Every worker has a fixed number of slots that control how many tasks it can run concurrently. You configure them with the slots option on the worker. If you set slots=5, the worker will run up to five tasks at the same time. Any additional tasks wait in the queue until a slot opens up.

Slots are a local limit - they protect the individual worker process from overcommitting its CPU, memory, or event loop. Concurrency controls are a global limit across your entire fleet - use them to prevent a single tenant or use-case from monopolizing capacity, or to respect the limits of an external resource like a third-party API or database connection pool. The two work together: concurrency controls decide how many runs Hatchet will allow to be active; slots decide how many of those runs each individual worker is willing to accept.

Choosing a slot count

Start with a slot count that matches the degree of parallelism your worker can sustain. For CPU-heavy tasks, that is typically the number of available cores. For I/O-heavy tasks (HTTP calls, database queries), you can safely go higher because most of the time is spent waiting.

Scaling workers

You can increase throughput in two ways: add more slots to a single worker, or run more worker processes. In most workloads, horizontal scaling (more workers) is the simplest path because each worker brings its own pool of slots and its own resources.

When running in Kubernetes or a similar orchestrator, you can autoscale workers based on queue depth using the Task Stats API. Hatchet also supports KEDA integration for event-driven autoscaling.

Task assignment

By default, Hatchet distributes tasks to any available worker that has registered the task. You can influence this behavior in several ways:

These are useful when a worker has specialized hardware (a GPU, a loaded ML model), or when co-locating related tasks on the same worker avoids redundant setup.

Running in production

In development, the fastest way to run a worker is hatchet worker dev, which handles authentication and hot-reloads your code on changes. In production, you’ll run workers as standalone processes or containers.

Workers and tasks

Workers and tasks have a many-to-many relationship. A single worker can register many tasks, and a single task can be registered on many workers. This means you can organize your workers by resource requirements, deployment boundary, or any other criterion - and Hatchet handles routing tasks to the right place.

If you haven’t already, read about tasks to understand how work is defined and configured.