Observability

AiHummer observability has two surfaces: the gateway serves a Prometheus GET /metrics endpoint you can scrape for the basics, and it can optionally push telemetry over OTLP to an OpenTelemetry endpoint you configure. Scrape /metrics for baseline monitoring; for traces and rich metrics point AiHummer at your OTLP collector and visualise the data with the bundled Grafana dashboards.

[!NOTE] pprof (/debug/pprof) is not exposed.

The Prometheus `/metrics` endpoint

GET /metrics serves Prometheus text-format metrics with no extra setup. Only non-sensitive gauges are exposed — build info, process uptime and runtime, DB connection-pool state and a readiness gauge; no tenant data, no secrets, no per-request labels. For traces and rich metrics, use the OTLP push.

OTLP push

Set a single variable to turn telemetry on:

# gateway.env — export telemetry to your OTLP collector
AIHUMMER_OTEL_ENDPOINT=http://otel-collector:4317

With AIHUMMER_OTEL_ENDPOINT set, the gateway pushes telemetry to that collector. From there, route it to your backend (Tempo, a metrics store, logs) and into Grafana.

Error reporting (Bugsink / Sentry)

The gateway ships a Sentry-protocol error-reporting client — it works with self-hosted Bugsink or any Sentry-compatible DSN. Set the DSN and panics and errors (including the boot phase and background workers) are reported to your tracker, tagged with the component and instance:

# gateway.env — error reporting (Bugsink or any Sentry-compatible DSN)
AIHUMMER_SENTRY_DSN=https://<key>@<your-bugsink-or-sentry>/<project>
AIHUMMER_SENTRY_ENVIRONMENT=production   # optional; defaults to production

Both knobs are settings-catalog entries (group Observability), so you can also set them from the web admin UI; the DB value takes precedence over env. Until a DSN is set, reporting is off.

Live logs in the admin UI

The admin UI’s “Logs” page is a live tail of the gateway journal: new lines are pulled in automatically every few seconds, with autoscroll while you are at the bottom. The toolbar has a line search and a level filter (all / errors / warnings / info / debug). Several other pages (Dashboard, Sessions, Channels) also refresh automatically, and long lists (audit, changes, notifications and so on) load page by page with a “Show more” button.

Full logs of every service still live in systemd — aihummer logs [unit] or journalctl -u aihummer-gateway.

Grafana dashboards

Ready-made Grafana dashboards ship with the release. Import them into your Grafana instance to get the operational views without building panels from scratch.

What to watch

These are the signals that tell you the system is healthy and turns are flowing:

Signal	Why it matters
Turn latency	End-to-end responsiveness of agent turns
Error rate	Failing turns / requests — the first sign of trouble
Delivery dispositions	Whether replies are actually reaching channels
Outbox / DLQ depth	Backlog of undelivered messages; a growing DLQ means delivery is stuck

A rising outbox or dead-letter-queue (DLQ) depth is the clearest early warning that delivery is backing up — watch it during rollouts and incidents.

System endpoints

Alongside OTLP, the gateway exposes small HTTP endpoints useful for probes, clocks and client diagnostics:

Method	Endpoint	Purpose
`GET`	`/metrics`	Prometheus metrics (build, runtime, DB pool, readiness)
`GET`	`/healthz`	Liveness + version
`GET`	`/readyz`	Readiness (checks Postgres; 503 if down)
`GET`	`/v1/ping`	Lightweight reachability check
`GET`	`/v1/time`	Server time
`POST`	`/v1/client-log`	Ingest client-side log events

The health and readiness probes are covered in detail under systemd & health checks.

Where to next

Probes and the production pre-flight checklist: systemd & health checks.
What to watch during a rolling upgrade: Upgrade policy.

The Prometheus /metrics endpoint