Prometheus Metrics for Self-Hosted Hatchet

This document provides an overview of the Prometheus metrics exposed by Hatchet, setup instructions for the metrics endpoint, and example PromQL queries to analyze them.

Setup

To enable Prometheus metrics for your Hatchet instance, you can set the following environment variables. The corresponding configuration YAML values are mentioned in parentheses. If you are deploying Hatchet in HA mode, these should be set on the grpc, controllers, and scheduler deployments.

• Required

  • SERVER_PROMETHEUS_ENABLED (prometheus.enabled)
    • Default: false
    • Description: Enables or disables the Prometheus metrics HTTP server.

• Optional

  • SERVER_PROMETHEUS_ADDRESS (prometheus.address)

    • Default: ":9090"
    • Description: The network address and port to bind the Prometheus metrics server to.

  • SERVER_PROMETHEUS_PATH (prometheus.path)

    • Default: "/metrics"
    • Description: The HTTP path at which metrics will be exposed.

Once enabled, you can setup any scraper that supports ingesting Prometheus metrics.

Tenant metrics endpoint

To enable the tenant API endpoint you can set the following environment variables:

• Required

  • SERVER_PROMETHEUS_SERVER_URL (prometheus.prometheusServerURL)
    • Description: The Prometheus server URL.

• Optional

  • SERVER_PROMETHEUS_SERVER_USERNAME (prometheus.prometheusServerUsername)

    • Description: The username to access the Prometheus instance via HTTP basic auth.

  • SERVER_PROMETHEUS_SERVER_PASSWORD (prometheus.prometheusServerPassword)

    • Description: The password to access the Prometheus instance via HTTP basic auth.

Example environment setup:

export SERVER_PROMETHEUS_ENABLED=true
export SERVER_PROMETHEUS_ADDRESS=":9999"
export SERVER_PROMETHEUS_PATH="/custom-metrics"

Restart your Hatchet server after setting these variables to apply the changes.

Global Metrics

The publish duration and transit histograms are labelled by kind (the pub/sub backend: rabbitmq, postgres, or nats) and topic_kind; hatchet_pubsub_publish_duration_seconds is additionally labelled by result. Two caveats when comparing backends:

• Publish duration is how long Pub blocks the caller, not how long the broker took to deliver, and only Postgres waits on the broker at all — it publishes with a query, while NATS buffers in memory and RabbitMQ returns once the frames hit the socket (publisher confirms are not enabled). Expect nats to report the smallest durations and postgres the largest, regardless of how quickly each actually delivers.
• Transit latency is a difference between two clocks, so it is subject to skew between the publishing and subscribing pods. Messages published by engines that predate the published_at stamp are not observed at all.

hatchet_pubsub_nats_scheduler_partition_drops_total exists only when NATS backend is enabled and covers the scheduler-partition subscription only. It is read from the NATS client's Subscription.Dropped() at scrape time. Drops happen when a subscription exceeds the nats.go default pending limits (500,000 messages / 64 MiB per subscription). For tenant-stream subscriptions the sampled nats pubsub async error warning log is the only drop signal.

hatchet_pubsub_stale_skipped_total is labelled by kind and topic_kind, and is currently reported by the NATS backend only. A NATS subscriber that falls behind (at least 100 messages waiting in the NATS client) skips messages whose published_at stamp is older than a fixed max age instead of handling them: 5 seconds for scheduler-partition (wake-ups that the scheduler's polling loops have already covered) and 30 seconds for tenant-stream (the same as the RabbitMQ backend's per-message TTL). Without a backlog no message is skipped, so clock skew between pods cannot cause skips on its own. A sustained non-zero rate means a subscriber cannot keep up. Skipped messages are not observed in hatchet_pubsub_transit_seconds, so read the two together. Pub/sub delivery is at most once and consumers also poll on an interval, so a skipped message is usually covered by the next poll. Two cases are not: a skipped wake-up for an idle queue waits for that queue's backed-off poll (up to 45 seconds), and a skipped task stream event is not resent. RabbitMQ's 30-second TTL instead expires tenant-stream events while they are still queued, not after delivery to a subscriber. Like transit latency, the age is subject to clock skew between pods.

The NATS backend runs each subscription's handlers concurrently, up to a fixed limit per subscription: 128 for scheduler-partition and 32 for tenant-stream. When a subscription reaches its limit, further deliveries wait in the NATS client's buffer (hatchet_pubsub_nats_scheduler_partition_pending_messages for the scheduler partition) until a handler returns. hatchet_pubsub_handlers_in_flight, hatchet_pubsub_handler_pool_full_total and hatchet_pubsub_handler_slot_wait_seconds are labelled by kind and topic_kind, and summed over a process's subscriptions of a topic kind. Transit latency is observed when a handler starts, so for NATS it includes the time spent waiting for a handler slot.

A steadily increasing hatchet_pubsub_handler_pool_full_total means handlers are too slow for the incoming rate. The limits are not configurable. Handlers are usually slow because they wait on the database, so more concurrent handlers would mostly add database load. Instead:

• Check database CPU and query latency.
• For scheduler-partition, run more scheduler instances. Each scheduler instance owns a partition and tenants are spread across the active partitions, so each subscription serves fewer tenants.
• For tenant-stream, run more grpc-api instances. A tenant-stream subscription calls the handlers of every client of that tenant connected to the instance, one after another, so spreading clients across instances shortens each call.

Example PromQL Queries

1. Rate of calls to the queuer method

rate(hatchet_queue_invocations_total[5m])

2. Average queue time in milliseconds

# Calculates average queue time over the past 5 minutes, converted to ms
rate(hatchet_queued_to_assigned_time_seconds_sum[5m])
  / rate(hatchet_queued_to_assigned_time_seconds_count[5m])
  * 1e3

3. Success and failure rates

rate(hatchet_succeeded_tasks_total[5m])
rate(hatchet_failed_tasks_total[5m])

4. Queue time distribution (histogram)

sum by (le) (
  rate(hatchet_queued_to_assigned_time_seconds_bucket[5m])
)

5. Rate of tasks created vs. retried

rate(hatchet_created_tasks_total[5m])
rate(hatchet_retried_tasks_total[5m])

6. Task Assignment Rate

rate(hatchet_assigned_tasks[5m])

7. Scheduling Timeout Rate

rate(hatchet_scheduling_timed_out[5m])

8. Rate Limiting Impact

rate(hatchet_rate_limited[5m])

9. Task Completion Ratio (Success vs Total)

rate(hatchet_succeeded_tasks_total[5m])
/
(rate(hatchet_succeeded_tasks_total[5m]) + rate(hatchet_failed_tasks_total[5m]))

10. Task Cancellation Rate

rate(hatchet_cancelled_tasks_total[5m])

11. Task Skip Rate

rate(hatchet_skipped_tasks_total[5m])

12. Queue Processing Efficiency (Assigned vs Created)

rate(hatchet_assigned_tasks[5m]) / rate(hatchet_created_tasks_total[5m])

13. Task Reassignment Rate

rate(hatchet_reassigned_tasks[5m])

Tenant Metrics

The hatchet_tenant_*_worker_label_slots metrics expose gauges for each unique worker label (key, value) pair and slot type, using the label_key, label_value, and slot_type Prometheus labels. A worker's slots count towards every label pair the worker carries, so a worker labeled pool=gpu, region=us-east contributes its slots to both the {label_key="pool", label_value="gpu"} and {label_key="region", label_value="us-east"} series.

The slot_type label separates the worker's slot pools (e.g. default and durable), which have independent capacities. Filter to the slot type you care about — usually default — when computing utilization; a worker's large durable slot pool would otherwise mask saturation of its default slots. Summing across slot_type values is safe (the pools are disjoint), unlike summing across label_key values.

Example PromQL Queries

1. Workflow Duration by Tenant and Status

rate(hatchet_tenant_workflow_duration_milliseconds_sum[5m])
by (tenant_id, workflow_name, status)
/
rate(hatchet_tenant_workflow_duration_milliseconds_count[5m])
by (tenant_id, workflow_name, status)

2. Tenant Queue Performance (95th percentile)

histogram_quantile(0.95,
  rate(hatchet_tenant_queued_to_assigned_time_seconds_bucket[5m])
) by (tenant_id)

3. Tenant Error Rate by Workflow

rate(hatchet_tenant_failed_tasks[5m]) by (tenant_id)
/
rate(hatchet_tenant_created_tasks[5m]) by (tenant_id)

4. Tenant Task Throughput

rate(hatchet_tenant_succeeded_tasks[5m]) by (tenant_id)

5. Tenant Retry Rate

rate(hatchet_tenant_retried_tasks[5m]) by (tenant_id)
/
rate(hatchet_tenant_created_tasks[5m]) by (tenant_id)

6. Workflow Duration Distribution by Tenant

sum by (tenant_id, le) (
  rate(hatchet_tenant_workflow_duration_milliseconds_bucket[5m])
)

7. Tenant Rate Limiting Impact

rate(hatchet_tenant_rate_limited[5m]) by (tenant_id)

8. Per-Tenant Queue Utilization

rate(hatchet_tenant_queue_invocations[5m]) by (tenant_id)

9. Tenant Scheduling Timeouts

rate(hatchet_tenant_scheduling_timed_out[5m]) by (tenant_id)

10. Tenant Task Assignment Success Rate

rate(hatchet_tenant_assigned_tasks[5m]) by (tenant_id)
/
rate(hatchet_tenant_created_tasks[5m]) by (tenant_id)

11. Tenant Task Reassignment Rate

rate(hatchet_tenant_reassigned_tasks[5m]) by (tenant_id)

12. Worker Slot Utilization by Label Pair

hatchet_tenant_used_worker_label_slots{label_key="pool", label_value="gpu", slot_type="default"}
/
hatchet_tenant_worker_label_slots{label_key="pool", label_value="gpu", slot_type="default"}

13. Queue Backlog by Additional Metadata Tag

sum(hatchet_tenant_additional_metadata_queue_size{key="prom_pool", value="gpu"}) or vector(0)

14. Queue Latency (p95) by Workflow

histogram_quantile(0.95,
  sum by (workflow_name, le) (
    rate(hatchet_tenant_queued_to_assigned_time_seconds_by_workflow_bucket[5m])
  )
)

The worker slot and queue size gauges are designed to drive autoscalers — see Autoscaling Workers for how to use them with KEDA.

Cross-Tenant Analysis

Example PromQL Queries

1. Top 5 Tenants by Task Volume

topk(5,
  sum by (tenant_id) (
    rate(hatchet_tenant_created_tasks[1h])
  )
)

2. Slowest Workflows Across All Tenants

topk(10,
  rate(hatchet_tenant_workflow_duration_milliseconds_sum[5m])
  /
  rate(hatchet_tenant_workflow_duration_milliseconds_count[5m])
) by (tenant_id, workflow_name)

3. Tenant Resource Consumption Comparison

sum by (tenant_id) (
  rate(hatchet_tenant_workflow_duration_milliseconds_sum[1h])
)
/ 1000 / 60  # Convert to minutes

Integration with Prometheus

This endpoint can be used to configure Prometheus to scrape tenant-specific metrics:

scrape_configs:
  - job_name: "hatchet-tenant-metrics"
    static_configs:
      - targets: ["cloud.onhatchet.run"]
    metrics_path: "/api/v1/tenants/707d0855-80ab-4e1f-a156-f1c4546cbf52/prometheus-metrics"
    scheme: "https"
    authorization:
      credentials: "your-api-token-here"

Note: Replace cloud.onhatchet.run with the URL where your Hatchet instance is hosted.

This provides tenant-isolated metrics that can be scraped directly by Prometheus or consumed by other monitoring tools that support the Prometheus text format.