Task Eviction

When a durable task enters a wait, whether from SleepFor, WaitForEvent, or WaitFor, Hatchet evicts the task from the worker. The worker slot is released, the task’s progress is persisted in the durable event log, and the task does not consume slots or hold resources while it is idle.

This is what makes durable tasks fundamentally different from regular tasks: a regular task consumes a slot for the entire duration of execution, even if it’s just sleeping. A durable task gives the slot back the moment it starts waiting.

How eviction works

1. Task reaches a wait. The durable task calls SleepFor, WaitForEvent, or WaitFor.
2. Checkpoint is written. Hatchet records the current progress in the durable event log.
3. Worker slot is freed. The task is evicted from the worker. The slot is immediately available for other tasks.
4. Wait completes. When the sleep expires or the expected event arrives, Hatchet re-queues the task.
5. Task resumes on any available worker. A worker picks up the task, replays the event log to the last checkpoint, and continues execution from where it left off.

The resumed task does not need to run on the same worker that originally started it. Any worker that has registered the task can pick it up.

Why eviction matters

Without eviction, a task that sleeps for 24 hours would consume a slot for the entire duration, wasting capacity that could be running other work. With eviction, the slot is freed immediately.

This is especially important for:

• Long waits — Tasks that sleep for hours or days should not hold slots.
• Human-in-the-loop — Waiting for a human to approve or respond could take minutes or weeks. Eviction ensures no resources are held in the meantime.
• Large fan-outs — A parent task that spawns thousands of children and waits for results can release its slot while the children run, preventing deadlocks where the parent holds resources that the children need.

Separate slot pools

Durable tasks consume slots from a separate slot pool than regular tasks. This prevents a common deadlock: if durable and regular tasks shared the same pool, a durable task waiting on child tasks could hold the very slot those children need to execute.

By isolating slot pools, Hatchet ensures that durable tasks waiting on children never starve the workers that need to run those children.

Eviction and determinism

Because a task may be evicted and resumed on a different worker at any time, the code between checkpoints must be deterministic. On resume, Hatchet replays the event log; it does not re-execute completed operations. If the code has changed between the original run and the replay, the checkpoint sequence may not match, leading to unexpected behavior.

No Eviction Needed

DAG tasks do not require eviction because they are never assigned to a worker until they can actually run. A worker slot is only allocated when all of the task’s conditions are met: parent tasks have completed, sleep durations have elapsed, and expected events have arrived.

This means resources are only consumed during active execution, never during waits.

How DAG scheduling works

1. Task is pending. The task exists in the workflow but is not queued. No worker slot is allocated. No resources are consumed.
2. Conditions are met. All parent tasks have completed, any sleep duration has elapsed, and any required events have arrived.
3. Task is queued. Only now does Hatchet place the task in the queue for worker assignment.
4. Task runs to completion. A worker picks up the task, executes it, and the slot is freed.

Why this matters

Because DAG tasks are only scheduled when ready, there is no wasted capacity:

• Sleep conditions — A task that waits 24 hours after its parent completes does not hold a slot. It sits in a pending state until the timer expires, then gets queued.
• Event conditions — A task waiting for an external event consumes no resources. When the event arrives, the task is queued and assigned a slot.
• Parent dependencies — Tasks waiting on upstream results are not queued until those results are available.

This is one of the advantages of DAGs: the scheduling model is simpler. You declare the conditions upfront, and Hatchet handles the timing. There is no eviction, no checkpointing, and no replay, because the task never starts until it’s ready to run straight through.