Skip to content

Prometheus and Grafana

Two ways into Grafana: metrics — counts, latencies, tokens and cost, scraped by Prometheus from the PrometheusAdapter — and traces — every span, sent over OTLP to Tempo or Grafana Cloud.

Metrics

Install e-track[prometheus].

from etrack.providers.adapters.sink.prometheus import PrometheusAdapter

async with PrometheusAdapter() as metrics:
    bus.subscribe(metrics)
    # mount the scrape app in your web app…
    app.mount("/metrics", metrics.app)
    # …or serve it on its own port
    async with GranianAdapter(metrics.app) as server:
        ...
Metric Labels
etrack_spans_total kind, name, status (ok, error) every closed span
etrack_span_seconds kind, name span latency, a histogram
etrack_runs_active runs in flight
etrack_tokens_total model, node, type (input, output, cache_read, cache_write, reasoning) model tokens, from model calls only
etrack_cost_usd_total model, node model cost, when the source or your Pricer knows it
etrack_fact_<name>_total the fact's labels one counter per typed fact → The run from inside

A span's name is its node, tool, subagent or model — bounded by your graph and your tools, never free text.

Setting default
ETRACK_PROMETHEUS_NAMESPACE etrack every metric's prefix
ETRACK_PROMETHEUS_BUCKETS [0.01 … 60] latency buckets, in seconds
# prometheus.yml
scrape_configs:
  - job_name: agents
    static_configs: [{targets: ["127.0.0.1:8000"]}]
    metrics_path: /metrics

Queries to start from: calls per second by tool sum by (name) (rate(etrack_spans_total{kind="tool"}[5m])); p95 model latency histogram_quantile(0.95, sum by (le, name) (rate(etrack_span_seconds_bucket{kind="llm"}[5m]))); spend per hour sum by (model) (increase(etrack_cost_usd_total[1h])); error ratio sum(rate(etrack_spans_total{status="error"}[5m])) / sum(rate(etrack_spans_total[5m])).

The registry is the adapter's own: two adapters never collide, and your app's metrics stay in yours.

Traces

Tempo and Grafana Cloud ingest OTLP: the OTLP driver with a basic auth.

ETRACK_OTLP_TARGETS='{"grafana": {
  "endpoint": "https://otlp-gateway-prod-eu-west-2.grafana.net/otlp",
  "auth": {"type": "basic", "username": "<instance id>", "password": "<token>"}}}'

Self-hosted Tempo: http://<host>:4318, no auth. → OpenTelemetry · Authentication