Skip to content

Metrics

Both the issuer and the verifier serve Prometheus metrics at GET /metrics: a count of the sessions they start and of the sessions they complete, by how each one ended, alongside the standard JVM and process metrics. The endpoint is served on the same port as the rest of the service's API.

The endpoint is off until you configure it, and a scrape must present a token of its own: API keys are not accepted.

Enabling the endpoint

Each service is enabled on its own: add a metrics block to the issuer's server.json, the verifier's, or both. A service without the block answers 404 on /metrics.

"metrics": {
  "scrapeTokens": {
    "Q2sx3xVwz9oRMSk1UQBnkZMIJQ7bbEz8T6vdYJkTB3CvYaL4bMyTfrUZzcXfGl8x": "Prometheus" // (1)!
  }
}
  1. Key: the token the scrape presents. Value: a label naming who it was issued to.

The block reads the same way as the API keys: the JSON key is the token, the value a label. It is read at startup, so adding or removing a token requires a restart of the service.

Scrape tokens and API keys are separate. An API key is refused at /metrics, and a scrape token is refused by the administrative endpoints, so the system that collects your metrics cannot start sessions or revoke credentials.

Managing scrape tokens

Treat them as you would API keys:

  • Issue one token per scraper, so you can revoke a single one without disturbing the others. That is what the label is for.
  • Generate tokens with a CSPRNG and make them long. The example above is 64 characters.
  • Keep them out of server.json in version control. They sit in the same file as your database password and API keys; see Going to production.
  • Revoking means removing the entry and restarting. There is no expiry.
  • Rotate without a gap in collection by listing both tokens for a while: add the new one and restart, move the scrape onto it, then remove the old one and restart again.

Scraping

Present the token as a bearer token. On the quickstart ports, 8079 for the issuer and 8081 for the verifier:

curl http://localhost:8079/metrics \
  -H 'Authorization: Bearer Q2sx3xVwz9oRMSk1UQBnkZMIJQ7bbEz8T6vdYJkTB3CvYaL4bMyTfrUZzcXfGl8x'
curl http://localhost:8081/metrics \
  -H 'Authorization: Bearer Q2sx3xVwz9oRMSk1UQBnkZMIJQ7bbEz8T6vdYJkTB3CvYaL4bMyTfrUZzcXfGl8x'

A missing Authorization header or an unrecognized token returns 401, with the reason in the body and a warning in the service's log. The scheme must be written exactly Bearer.

The format follows the request's Accept header. A scrape that asks for OpenMetrics, as Prometheus does by default, gets OpenMetrics 1.0; anything else, including curl, gets the Prometheus text format. The protobuf format is not offered.

In Prometheus, give each service a job of its own and read the token from a file rather than writing it into the configuration:

scrape_configs:
    - job_name: issuer
      authorization:
          credentials_file: /etc/prometheus/issuer-scrape-token
      static_configs:
          - targets: ['issuer-backend:8079']
    - job_name: verifier
      authorization:
          credentials_file: /etc/prometheus/verifier-scrape-token
      static_configs:
          - targets: ['verifier-backend:8081']

/metrics sits on the same port as the endpoints wallets call, so it is reachable wherever they are. Scrape the services on their internal addresses, and do not route /metrics through the proxy that exposes them to the internet. See Going to production.

Session metrics

Metric Labels Counts
issuer_sessions_started_total tenant Issuance sessions started
issuer_sessions_completed_total tenant, status Issuance sessions that reached a final status
verifier_sessions_started_total tenant Verification sessions started
verifier_sessions_completed_total tenant, status Verification sessions that reached a final status

tenant is empty on a self-hosted deployment. status is the status the session ended in, as reported by the service's session status endpoint.

An issuance session is started by POST /issuance/oid4vci/new-session. A verification session is started by POST /presentation/oid4vp/new-session, and also each time a wallet fetches a static presentation: every holder who scans the same QR code gets a session of their own.

On the issuer, a session ends in one of:

status What it means
ISSUANCE_SUCCEEDED The credential was issued and handed to the wallet, after your callback accepted it if one is set.
ISSUANCE_FAILED The credential could not be issued — for example, the wallet's proof did not verify.
DELIVERY_REJECTED The credential was signed, but your callback answered with a non-successful HTTP response.
DELIVERY_FAILED The credential was signed, but your callback could not be reached at all.

In both delivery statuses the wallet is refused the credential: ISSUANCE_SUCCEEDED is the only status in which a holder received one.

On the verifier, a session ends in VERIFICATION_SUCCEEDED, VERIFICATION_FAILED, WALLET_REJECTED, DELIVERY_REJECTED or DELIVERY_FAILED, with the meanings given in Request a presentation.

On both services, a request turned away before it reaches a session — an unknown session, or one that has already ended — changes neither counter.

Reading the counters

  • They start from zero when the service restarts. The counts are kept in memory, not in the database. rate() and increase() account for the reset; a raw value does not.
  • A series appears on its first count. Until a status has occurred, there is no series for it rather than one at 0, so a query for a status that never happened returns nothing.
  • Add up across instances. A session can start on one instance and end on another, so compare started and completed sessions only once summed over every instance of the service.
  • Started minus completed is not the number in progress. A session the holder never finishes — a QR code nobody scans, a wallet closed without answering — never reaches a final status, so the difference keeps the abandoned sessions too.
  • A count is not proof of a stored session. Counters are incremented before the request's transaction commits, so a request that fails after that point is still counted.

Example queries

The share of verifications that succeeded over the last hour:

sum(increase(verifier_sessions_completed_total{status="VERIFICATION_SUCCEEDED"}[1h]))
/
sum(increase(verifier_sessions_completed_total[1h]))

Results that could not be handed to your application, on either service — worth an alert, since in each case the holder did their part and your application never received the outcome:

sum by (job, status) (
  increase({__name__=~"(issuer|verifier)_sessions_completed_total", status=~"DELIVERY_.*"}[15m])
) > 0

JVM and process metrics

Each service also exposes the standard JVM metrics of the Prometheus Java client, covering:

  • Memory: heap and non-heap use, per memory pool — jvm_memory_used_bytes, jvm_memory_max_bytes, jvm_memory_pool_used_bytes.
  • Garbage collection: time spent per collector — jvm_gc_collection_seconds.
  • Threads: current, peak and deadlocked threads, and threads by state — jvm_threads_current, jvm_threads_state.
  • Classes, buffer pools and compilation — jvm_classes_currently_loaded, jvm_buffer_pool_used_bytes, jvm_compilation_time_seconds_total.
  • The runtime: Java version and vendor — jvm_runtime_info.
  • The process: CPU time, memory, open file descriptors and start time — process_cpu_seconds_total, process_resident_memory_bytes, process_open_fds, process_start_time_seconds.

The services do not measure HTTP requests or the database. For request rates and latencies, measure at the proxy in front of them; for whether the database is reachable, use /readyz.

What's Next