Rate Limiting Step Runs in Hatchet

Hatchet allows you to enforce rate limits on step runs in your workflows, enabling you to control the rate at which workflow runs consume resources, such as external API calls, database queries, or other services. By defining rate limits, you can prevent step runs from exceeding a certain number of requests per time window (e.g., per second, minute, or hour), ensuring efficient resource utilization and avoiding overloading external services.

The state of active rate limits can be viewed in the dashboard in the Rate Limit resource tab.

Dynamic vs Static Rate Limits

Hatchet offers two patterns for Rate Limiting step runs:

1. Dynamic Rate Limits: Allows for complex rate limiting scenarios, such as per-user limits, by using input or additional_metadata keys to upsert a limit at runtime.
2. Static Rate Limits: Allows for simple rate limiting for resources known prior to runtime (e.g., external APIs).

Dynamic Rate Limits

Dynamic rate limits are ideal for complex scenarios where rate limits need to be partitioned by resources that are only known at runtime.

This pattern is especially useful for:

1. Rate limiting individual users or tenants
2. Implementing variable rate limits based on subscription tiers or user roles
3. Dynamically adjusting limits based on real-time system load or other factors

How It Works

1. Define the dynamic rate limit key with a CEL (Common Expression Language) Expression on the key, referencing either input or additional_metadata.
2. Provide this key as part of the workflow trigger or event input or additional_metadata at runtime.
3. Hatchet will create or update the rate limit based on the provided key and enforce it for the step run.

Declaring and Consuming Dynamic Rate Limits

@hatchet.step(rate_limits=[
  RateLimit(
    dynamic_key='input.user_id',
    units=1,
    limit=10,
    duration=RateLimitDuration.MINUTE,
    )])
def step1(self, context: Context):
    print("executed step1")
    pass

Static Rate Limits

Static Rate Limits (formerly known as Global Rate Limits) are defined as part of your worker startup lifecycle prior to runtime. This model provides a single "source of truth" for pre-defined resources such as:

1. External API resources that have a rate limit across all users or tenants
2. Database connection pools with a maximum number of concurrent connections
3. Shared computing resources with limited capacity

How It Works

1. Declare static rate limits using the put_rate_limit method in the Admin client before starting your worker.
2. Specify the units of consumption for a specific rate limit key in each step definition using the rate_limits configuration.
3. Hatchet enforces the defined rate limits by tracking the number of units consumed by each step run across all workflow runs.

If a step run exceeds the rate limit, Hatchet re-queues the step run until the rate limit is no longer exceeded.

Declaring Static Limits

Define the static rate limits that can be consumed by any step run across all workflow runs using the put_rate_limit method in the Admin client within your code.

hatchet.admin.put_rate_limit('example-limit', 10, RateLimitDuration.MINUTE)

Consuming Static Rate Limits

With your rate limit key defined, specify the units of consumption for a specific key in each step definition by adding the rate_limits configuration to your step definition in your workflow.

@hatchet.step(rate_limits=[RateLimit(key='example-limit', units=1)])
def step1(self, context: Context):
    print("executed step1")
    pass

Limiting Workflow Runs

To rate limit an entire workflow run, it's recommended to specify the rate limit configuration on the entry step (i.e., the first step in the workflow). This will gate the execution of all downstream steps in the workflow.