Frequently Asked Questions

This page provides answers to a number of the most common questions we're asked, to help you keep making great use of Hatchet!

How do I choose how many slots to set on my worker?

The default slot count for workers in Hatchet is 100. In many cases, leaving the default as-is will be perfectly fine, especially when first getting set up with Hatchet.

Over time, you'll likely run into one of two issues: Resource starvation (meaning the worker is using up too much memory, CPU, etc.), or wanting to squeeze more juice out of your workers.

If your workers are resource starved, there are basically two options:

1. Reduce the slot count, so the worker runs less work concurrently. This is a blunt instrument, in the sense that it doesn't let you tune resources to the needs of the workload running on the worker. For instance, if you're using 100% of your memory but only 10% of your CPU, reducing the slot count will likely help the worker stay online, but you'll be significantly under-utilizing CPU. In this case, you can:
2. Reconfigure the specs of the machine the worker is running on. For instance, in the example above, you might be able to migrate from a CPU-optimized machine to a memory-optimized one, which will give you more efficient resource utilization across the board.

If only some of your tasks are heavy, task slot cost lets those tasks consume more than one slot, so the slot count can stay tuned for the light ones.

On the other hand, if your workers are underutilizing resources, your options are:

1. Increase the number of slots on them so they can pick up more work. This is especially helpful for heavily I/O bound tasks, which generally are spending most of their time waiting.
2. Similar to the opposite case of resource starvation, you can scale down the resource requirements of the machine the worker is running on.

Why am I seeing missed heartbeats and task reassignments?

Hatchet uses heartbeats to monitor worker health. Workers send a heartbeat every 4 seconds. If the engine does not receive a heartbeat for 30 seconds, the engine considers the worker to be inactive, and re-queues its in-flight tasks for other workers to pick up.

There are a number of common reasons a worker might miss heartbeats:

• Process crash - the worker process exits unexpectedly (OOM kill, unhandled exception, SIGKILL).
• Network disruption - the connection between the worker and the Hatchet engine is interrupted (DNS failure, firewall change, cloud network blip).
• Resource pressure - High CPU or memory usage can starve the worker for resources

What happens to running tasks when a worker shuts down?

When a worker begins to shut down, a few things happen:

1. The worker receives a shutdown signal from the box it's running on (e.g. a SIGTERM) indicating that it should begin shutting down.
2. When the worker receives this signal, it sends a message to the engine to put itself into a PAUSED state. A worker in a PAUSED state no longer receives new tasks from the engine.
3. The now-paused worker will continue to run to allow already-running tasks to drain until it reaches the end of its shutdown grace period.
4. If the worker reaches the end of the grace period and it has not yet drained, it will (usually) be forcibly killed via a signal like SIGKILL. If this happens, the worker is forcibly shut down, causing it to stop heartbeating.
5. Once the worker has stopped heartbeating, the engine declares it dead, and will reassign any tasks that were running on it to a new worker.

This means that there are some practical considerations here:

• Ideally, tasks would be idempotent, because a reassign could result in work being executed more than once (note that this is a good idea in virtually any queueing system).
• It's often advantageous to try to keep tasks relatively short if possible, so that they can complete in the grace period the worker is given in order to avoid reassignments.

faq, heartbeat, reassignment, reassign, slots, shutdown