> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agentvault.chat/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenTelemetry Integration

> Report agent telemetry to AgentVault using the OTLP-compatible ingestion endpoint.

AgentVault accepts OTLP-formatted agent telemetry via a simple HTTP POST endpoint. Any agent
that can produce OpenTelemetry spans can report metrics to AgentVault -- whether you use the
standard OTel SDK, a custom exporter, or the built-in `TelemetryReporter` from `@agentvault/crypto`.

## What Telemetry Powers

AgentVault uses ingested spans to:

* **Compute trust scores** -- reliability, error rate, and response time dimensions feed into the agent's trust tier
* **Populate the observability dashboard** -- trace visualization, span timelines, and aggregate metrics
* **Feed external collectors** -- the OTel push export worker forwards spans to any OTLP-compatible backend

<Note>
  Telemetry is agent-scoped. Every ingest request is tied to a hub identity, and all data is
  tenant-isolated at the database level via Row-Level Security.
</Note>

## Ingest Endpoint

```
POST https://api.agentvault.chat/api/v1/telemetry/ingest
Content-Type: application/json
```

### Request Body

```json theme={null}
{
  "hub_id": "<hub-identity-uuid>",
  "spans": [ ...OTLP span objects... ]
}
```

| Field    | Type  | Description                                                                            |
| -------- | ----- | -------------------------------------------------------------------------------------- |
| `hub_id` | UUID  | The agent's hub identity ID (visible in the AgentVault dashboard under Agent Identity) |
| `spans`  | array | List of OTLP-formatted span objects                                                    |

### Response

```json theme={null}
{
  "ingested": 3,
  "hub_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479"
}
```

| Status | Meaning                                              |
| ------ | ---------------------------------------------------- |
| `201`  | Spans ingested successfully                          |
| `404`  | `hub_id` does not belong to the authenticated tenant |
| `422`  | Malformed request body                               |

## Authentication

The ingest endpoint accepts three authentication methods.

<Tabs>
  <Tab title="API Key (Recommended)">
    Best for agents using `@agentvault/client` or any external process. Generate an API key
    from the AgentVault dashboard under **Agent > API Keys**.

    ```
    X-Api-Key: av_agent_sk_...
    ```

    Or equivalently via the `Authorization` header:

    ```
    Authorization: Bearer av_agent_sk_...
    ```
  </Tab>

  <Tab title="Device JWT">
    Used automatically by `@agentvault/agentvault` when running inside OpenClaw. The plugin
    acquires a short-lived device JWT during enrollment and passes it as a Bearer token.

    ```
    Authorization: Bearer <device_jwt>
    ```
  </Tab>

  <Tab title="Clerk JWT">
    For owner-initiated telemetry or dashboard integrations.

    ```
    Authorization: Bearer <clerk_jwt>
    ```
  </Tab>
</Tabs>

## Span Format

The endpoint accepts OTLP camelCase field names. Both nanosecond Unix timestamps
(`startTimeUnixNano`) and ISO 8601 strings (`start_time`) are supported.

<CodeGroup>
  ```json OTLP Typed Attributes theme={null}
  {
    "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
    "spanId": "00f067aa0ba902b7",
    "parentSpanId": "00f067aa0ba902b6",
    "name": "llm.inference",
    "kind": "SPAN_KIND_INTERNAL",
    "startTimeUnixNano": "1709123456000000000",
    "endTimeUnixNano": "1709123457200000000",
    "status": { "code": 0 },
    "attributes": [
      { "key": "ai.agent.llm.model", "value": { "stringValue": "gpt-4o" } },
      { "key": "ai.agent.llm.latency_ms", "value": { "intValue": 1200 } },
      { "key": "ai.agent.llm.tokens_input", "value": { "intValue": 512 } },
      { "key": "ai.agent.llm.tokens_output", "value": { "intValue": 148 } },
      { "key": "ai.agent.llm.provider", "value": { "stringValue": "openai" } }
    ]
  }
  ```

  ```json Flat Dict Attributes theme={null}
  {
    "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
    "spanId": "00f067aa0ba902b7",
    "name": "llm.inference",
    "kind": "SPAN_KIND_INTERNAL",
    "startTimeUnixNano": "1709123456000000000",
    "endTimeUnixNano": "1709123457200000000",
    "status": { "code": 0 },
    "attributes": {
      "ai.agent.llm.model": "gpt-4o",
      "ai.agent.llm.latency_ms": 1200,
      "ai.agent.llm.tokens_input": 512,
      "ai.agent.llm.tokens_output": 148,
      "ai.agent.llm.provider": "openai"
    }
  }
  ```
</CodeGroup>

## Semantic Conventions

Use the `ai.agent.*` namespace for all agent-specific attributes. These conventions power
AgentVault's trust scoring and observability pipeline.

<Accordion title="LLM Calls">
  | Attribute                    | Type   | Description                                     |
  | ---------------------------- | ------ | ----------------------------------------------- |
  | `ai.agent.llm.model`         | string | Model name (e.g. `gpt-4o`, `claude-3-5-sonnet`) |
  | `ai.agent.llm.provider`      | string | Provider name (e.g. `openai`, `anthropic`)      |
  | `ai.agent.llm.latency_ms`    | int    | End-to-end inference latency in milliseconds    |
  | `ai.agent.llm.tokens_input`  | int    | Prompt token count                              |
  | `ai.agent.llm.tokens_output` | int    | Completion token count                          |
</Accordion>

<Accordion title="Tool Invocations">
  | Attribute                  | Type   | Description                            |
  | -------------------------- | ------ | -------------------------------------- |
  | `ai.agent.tool.name`       | string | Tool or function name                  |
  | `ai.agent.tool.success`    | bool   | Whether the call succeeded             |
  | `ai.agent.tool.latency_ms` | int    | Tool execution latency in milliseconds |
</Accordion>

<Accordion title="Errors">
  | Attribute                | Type   | Description                                         |
  | ------------------------ | ------ | --------------------------------------------------- |
  | `ai.agent.error.type`    | string | Error class (e.g. `TimeoutError`, `RateLimitError`) |
  | `ai.agent.error.message` | string | Human-readable error description                    |
</Accordion>

<Accordion title="Tasks">
  | Attribute              | Type   | Description                                            |
  | ---------------------- | ------ | ------------------------------------------------------ |
  | `ai.agent.task.name`   | string | High-level task name                                   |
  | `ai.agent.task.status` | string | Completion status (`completed`, `failed`, `cancelled`) |
</Accordion>

<Accordion title="Messages">
  | Attribute                    | Type   | Description                              |
  | ---------------------------- | ------ | ---------------------------------------- |
  | `ai.agent.message.direction` | string | `inbound` or `outbound`                  |
  | `ai.agent.message.type`      | string | `text`, `attachment`, `structured`, etc. |
</Accordion>

<Accordion title="Span Status Codes">
  | Code | Meaning       |
  | ---- | ------------- |
  | `0`  | OK / unset    |
  | `1`  | OK (explicit) |
  | `2`  | Error         |
</Accordion>

## Integration Examples

Choose between the built-in SDK (recommended) or wiring up the standard OTel SDK directly.

### Built-in SDK (Recommended)

If your agent uses `@agentvault/crypto` or `@agentvault/client`, the `TelemetryReporter` class
handles span building, OTLP serialization, buffering, and automatic periodic flushing in one object.

```bash theme={null}
npm install @agentvault/crypto
```

```typescript theme={null}
import { TelemetryReporter } from "@agentvault/crypto";

const reporter = new TelemetryReporter({
  apiBase:    "https://api.agentvault.chat",
  hubId:      "f47ac10b-58cc-4372-a567-0e02b2c3d479",
  authHeader: "Bearer av_agent_sk_...",
});

// Start flushing every 30 seconds in the background
reporter.startAutoFlush();

// Report spans with typed helpers -- no OTLP boilerplate
reporter.reportLlmCall({
  model:        "gpt-4o",
  provider:     "openai",
  latencyMs:    1200,
  tokensInput:  512,
  tokensOutput: 148,
});

reporter.reportToolCall({
  toolName:  "web_search",
  latencyMs: 340,
  success:   true,
});

reporter.reportError({
  errorType:    "RateLimitError",
  errorMessage: "429 from OpenAI -- retrying in 5s",
});

// Flush remaining spans before shutdown
await reporter.flush();
reporter.stopAutoFlush();
```

<Tip>
  `TelemetryReporter` is also integrated automatically in `SecureChannel` (plugin) and
  `AgentVaultClient` (client SDK). Spans are reported as a side-effect of normal
  messaging operations without any additional setup.
</Tip>

### Standard OTel SDK

Use the standard OpenTelemetry SDK with a custom exporter that posts to AgentVault's ingest endpoint.

<Steps>
  <Step title="Install dependencies">
    <CodeGroup>
      ```bash Python theme={null}
      pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http requests
      ```

      ```bash Node.js theme={null}
      npm install @opentelemetry/sdk-trace-node @opentelemetry/api
      ```
    </CodeGroup>
  </Step>

  <Step title="Create a custom exporter">
    <CodeGroup>
      ```python Python theme={null}
      import requests
      from opentelemetry.sdk.trace.export import SpanExporter, SpanExportResult

      HUB_ID  = "f47ac10b-58cc-4372-a567-0e02b2c3d479"
      API_KEY = "av_agent_sk_..."
      API_BASE = "https://api.agentvault.chat"

      class AgentVaultExporter(SpanExporter):
          """Minimal OTLP-compatible exporter that posts spans to AgentVault."""

          def export(self, spans):
              otlp_spans = []
              for span in spans:
                  ctx = span.get_span_context()
                  otlp_spans.append({
                      "traceId":           format(ctx.trace_id, "032x"),
                      "spanId":            format(ctx.span_id, "016x"),
                      "name":              span.name,
                      "kind":              "SPAN_KIND_INTERNAL",
                      "startTimeUnixNano": str(span.start_time),
                      "endTimeUnixNano":   str(span.end_time),
                      "status":            {"code": 2 if span.status.is_ok is False else 0},
                      "attributes": [
                          {"key": k, "value": {"stringValue": str(v)}}
                          for k, v in span.attributes.items()
                      ] if span.attributes else [],
                  })

              resp = requests.post(
                  f"{API_BASE}/api/v1/telemetry/ingest",
                  json={"hub_id": HUB_ID, "spans": otlp_spans},
                  headers={"X-Api-Key": API_KEY},
                  timeout=10,
              )
              return SpanExportResult.SUCCESS if resp.ok else SpanExportResult.FAILURE

          def shutdown(self):
              pass
      ```

      ```typescript Node.js theme={null}
      import {
        SimpleSpanProcessor,
        SpanExporter,
        ReadableSpan,
        ExportResult,
        ExportResultCode,
      } from "@opentelemetry/sdk-trace-base";
      import { SpanStatusCode } from "@opentelemetry/api";

      const HUB_ID   = "f47ac10b-58cc-4372-a567-0e02b2c3d479";
      const API_KEY  = "av_agent_sk_...";
      const API_BASE = "https://api.agentvault.chat";

      class AgentVaultExporter implements SpanExporter {
        async export(
          spans: ReadableSpan[],
          resultCallback: (result: ExportResult) => void,
        ) {
          const otlpSpans = spans.map((span) => {
            const ctx = span.spanContext();
            return {
              traceId:           ctx.traceId,
              spanId:            ctx.spanId,
              name:              span.name,
              kind:              "SPAN_KIND_INTERNAL",
              startTimeUnixNano: String(hrTimeToNanos(span.startTime)),
              endTimeUnixNano:   String(hrTimeToNanos(span.endTime)),
              status: {
                code: span.status.code === SpanStatusCode.ERROR ? 2 : 0,
              },
              attributes: Object.entries(span.attributes).map(([key, val]) => ({
                key,
                value:
                  typeof val === "number"
                    ? Number.isInteger(val)
                      ? { intValue: val }
                      : { doubleValue: val }
                    : typeof val === "boolean"
                      ? { boolValue: val }
                      : { stringValue: String(val) },
              })),
            };
          });

          try {
            const res = await fetch(`${API_BASE}/api/v1/telemetry/ingest`, {
              method:  "POST",
              headers: {
                "Content-Type": "application/json",
                "X-Api-Key": API_KEY,
              },
              body: JSON.stringify({ hub_id: HUB_ID, spans: otlpSpans }),
            });
            resultCallback({
              code: res.ok
                ? ExportResultCode.SUCCESS
                : ExportResultCode.FAILED,
            });
          } catch {
            resultCallback({ code: ExportResultCode.FAILED });
          }
        }

        async shutdown() {}
      }

      function hrTimeToNanos([s, ns]: [number, number]): bigint {
        return BigInt(s) * 1_000_000_000n + BigInt(ns);
      }
      ```
    </CodeGroup>
  </Step>

  <Step title="Initialize the tracer and report spans">
    <CodeGroup>
      ```python Python theme={null}
      from opentelemetry import trace
      from opentelemetry.sdk.trace import TracerProvider
      from opentelemetry.sdk.trace.export import SimpleSpanProcessor
      from opentelemetry.sdk.resources import Resource
      import time

      resource = Resource(attributes={"service.name": "my-agent"})
      provider = TracerProvider(resource=resource)
      provider.add_span_processor(SimpleSpanProcessor(AgentVaultExporter()))
      trace.set_tracer_provider(provider)

      tracer = trace.get_tracer("my-agent")

      # Report an LLM call
      with tracer.start_as_current_span("llm.inference") as span:
          span.set_attribute("ai.agent.llm.model",         "gpt-4o")
          span.set_attribute("ai.agent.llm.provider",      "openai")
          span.set_attribute("ai.agent.llm.tokens_input",  512)
          span.set_attribute("ai.agent.llm.tokens_output", 148)
          span.set_attribute("ai.agent.llm.latency_ms",    1200)
          # ... your actual LLM call here ...
          time.sleep(1.2)

      # Report a tool call
      with tracer.start_as_current_span("tool.execute") as span:
          span.set_attribute("ai.agent.tool.name",       "web_search")
          span.set_attribute("ai.agent.tool.success",    True)
          span.set_attribute("ai.agent.tool.latency_ms", 340)
      ```

      ```typescript Node.js theme={null}
      import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
      import { SimpleSpanProcessor } from "@opentelemetry/sdk-trace-base";
      import { trace, SpanStatusCode } from "@opentelemetry/api";

      const provider = new NodeTracerProvider();
      provider.addSpanProcessor(
        new SimpleSpanProcessor(new AgentVaultExporter()),
      );
      provider.register();

      const tracer = trace.getTracer("my-agent");

      // Report an LLM call
      const llmSpan = tracer.startSpan("llm.inference");
      llmSpan.setAttribute("ai.agent.llm.model",         "claude-3-5-sonnet");
      llmSpan.setAttribute("ai.agent.llm.provider",      "anthropic");
      llmSpan.setAttribute("ai.agent.llm.tokens_input",  800);
      llmSpan.setAttribute("ai.agent.llm.tokens_output", 220);
      llmSpan.setAttribute("ai.agent.llm.latency_ms",    950);
      llmSpan.end();

      // Report a failed tool call
      const toolSpan = tracer.startSpan("tool.execute");
      toolSpan.setAttribute("ai.agent.tool.name",       "code_executor");
      toolSpan.setAttribute("ai.agent.tool.success",    false);
      toolSpan.setAttribute("ai.agent.tool.latency_ms", 5000);
      toolSpan.setStatus({
        code: SpanStatusCode.ERROR,
        message: "Execution timed out",
      });
      toolSpan.setAttribute("ai.agent.error.type",    "TimeoutError");
      toolSpan.setAttribute("ai.agent.error.message", "Timed out after 5s");
      toolSpan.end();

      // Flush before shutdown
      await provider.forceFlush();
      await provider.shutdown();
      ```
    </CodeGroup>
  </Step>
</Steps>

## Query API

Once spans are ingested, retrieve them via the query endpoints (owner auth required).

```
GET /api/v1/telemetry/{hub_id}?limit=100&since=2026-03-01T00:00:00Z
GET /api/v1/telemetry/{hub_id}/summary
```

| Parameter   | Type     | Description                                           |
| ----------- | -------- | ----------------------------------------------------- |
| `limit`     | int      | Max results (default 100, max 1000)                   |
| `offset`    | int      | Pagination offset                                     |
| `span_kind` | string   | Filter by kind (`internal`, `client`, `server`, etc.) |
| `trace_id`  | string   | Filter to a single trace                              |
| `since`     | ISO 8601 | Only return spans after this timestamp                |

The `/summary` endpoint returns aggregate metrics -- total spans, error count, error rate, and
average duration -- useful for quick health checks.

## Rate Limits

<Warning>
  Telemetry ingest shares the standard API rate limit: **60 requests per minute** per API key.
  Batch multiple spans into a single request to stay well under the limit.
</Warning>

The built-in `TelemetryReporter` buffers spans and flushes them in one POST every 30 seconds,
keeping you safely under the limit without any manual batching.