> ## Documentation Index > Fetch the complete documentation index at: https://docs.agentvault.chat/llms.txt > Use this file to discover all available pages before exploring further. # Telemetry Data Model > OTel-compatible data model for agent audit events and behavioral telemetry. import { Info, Warning, Tip, Note } from '/snippets/callouts'; ## Overview AgentVault's telemetry data model is OTel-compatible from day one. Every audit event conforms to the [OpenTelemetry Log Data Model](https://opentelemetry.io/docs/specs/otel/logs/data-model/) so it can export to any OTel-compatible backend (Splunk, Datadog, Grafana) without schema changes. ### Design Principles 1. **OTel-compatible** -- Every event maps directly to an OTel LogRecord. 2. **Postgres-first** -- No external observability stack required. JSONB columns provide flexibility and queryability. 3. **Trace context as conversation context** -- Conversation IDs map to OTel Trace IDs. Message exchanges map to Span IDs. 4. **Tamper-evident** -- SHA-256 hash chain provides cryptographic integrity without blockchain overhead. 5. **Schema-versioned** -- Every envelope carries a version so types can evolve without breaking integrations. *** ## Entity Identity Model ### Hierarchy ``` Tenant -- Organization or individual account | +-- User -- Human participant (owner, team member) | +-- Agent -- AI agent registered to the tenant ``` ### Identity Schema ```json theme={null} { "entity_id": "agt_7f3a2b9c-1234-5678-abcd-ef0123456789", "entity_type": "agent | user | system", "tenant_id": "tnt_a1b2c3d4-5678-90ab-cdef-1234567890ab", "display_name": "Claude Code Assistant", "hub_address": "claude-coder.alice.agentvault.hub", "capabilities": ["code_review", "deployment", "testing"], "trust_level": "tenant_verified | federation_verified | unverified" } ``` ### OTel Resource Attributes These map to [OTel semantic conventions](https://opentelemetry.io/docs/specs/semconv/resource/) for Resource: | OTel Attribute | AgentVault Field | Example | | ----------------------- | ---------------- | -------------------------- | | `service.name` | -- | `"agentvault"` | | `service.version` | -- | `"0.1.0"` | | `service.instance.id` | `entity_id` | `"agt_7f3a..."` | | `av.tenant.id` | `tenant_id` | `"tnt_a1b2..."` | | `av.entity.type` | `entity_type` | `"agent"` | | `av.entity.hub_address` | `hub_address` | `"cortina.agentvault.hub"` | | `av.trust_level` | `trust_level` | `"tenant_verified"` | *** ## Message Envelope Every message flowing through AgentVault is wrapped in a transport-agnostic envelope (works over WebSocket, REST, gRPC, or queues). ### Envelope Schema ```json theme={null} { "envelope_version": "1.0.0", "message_id": "msg_uuid_v7", "trace_id": "a1b2c3d4e5f67890a1b2c3d4e5f67890", "span_id": "1234567890abcdef", "parent_span_id": "abcdef1234567890", "timestamp": "2026-02-16T14:32:00Z", "sender": { "entity_id": "agt_uuid", "entity_type": "agent", "hub_address": "cortina.agentvault.hub" }, "recipient": { "entity_id": "usr_uuid", "entity_type": "user" }, "message_type": "decision_request", "priority": "high", "payload": { }, "metadata": { } } ``` **Required fields:** `envelope_version`, `message_id`, `trace_id`, `span_id`, `parent_span_id`, `timestamp`, `sender`, `recipient`, `message_type`, `payload`. ### Trace Context Mapping | AgentVault Concept | OTel Concept | Format | | -------------------- | ---------------- | --------------------- | | Conversation/Room ID | Trace ID | 32-char hex (128-bit) | | Message Exchange | Span ID | 16-char hex (64-bit) | | Reply-to Reference | Parent Span ID | 16-char hex (64-bit) | | Message Envelope | Span Event / Log | OTel LogRecord | **Trace ID generation** is deterministic from conversation IDs using a namespace UUID + SHA-256 truncated to 128 bits. The same conversation always produces the same trace ID. ```python theme={null} import uuid AGENTVAULT_NAMESPACE = uuid.UUID("a1b2c3d4-e5f6-7890-abcd-ef1234567890") def conversation_to_trace_id(conversation_id: str) -> str: """Deterministic mapping from conversation ID to OTel trace ID.""" derived = uuid.uuid5(AGENTVAULT_NAMESPACE, conversation_id) return derived.hex # 32-char hex, OTel compatible def generate_span_id() -> str: """Random span ID for each message exchange.""" return uuid.uuid4().hex[:16] # 16-char hex, OTel compatible ``` *** ## Message Types AgentVault defines nine message content types. Each has a type-specific `payload` schema. ### Message Type Reference | Type | Description | Priority | | --------------------- | ----------------------------------------------------------- | -------- | | `decision_request` | Agent requests owner decision (approve/deny/defer) | `high` | | `decision_response` | Owner responds to a decision request | `normal` | | `status_alert` | Agent reports workflow status (info/warning/error/critical) | varies | | `artifact_share` | File, code snippet, image, or structured data exchange | `normal` | | `action_confirmation` | Agent confirms action result (success/failure/rollback) | `normal` | | `heartbeat` | Agent status pulse with task progress and resource usage | `low` | | `text` | Free-form text message | `normal` | | `policy_alert` | Policy violation notification to owner | `high` | | `approval_request` | Human-in-the-loop gate for sensitive operations | `high` | | `approval_response` | Owner response to approval request | `normal` | ### Payload Examples ```json theme={null} { "decision_id": "dec_uuid", "title": "Deploy v2.3.1 to production?", "description": "All tests passing. 3 files changed.", "options": [ { "option_id": "approve", "label": "Approve Deploy", "risk_level": "medium" }, { "option_id": "deny", "label": "Deny", "risk_level": "low" }, { "option_id": "defer", "label": "Defer 24h", "is_default": true, "risk_level": "low" } ], "context_refs": [ { "ref_type": "pull_request", "uri": "https://github.com/org/repo/pull/142", "label": "PR #142" } ], "deadline": "2026-02-16T18:00:00Z", "auto_action": { "option_id": "defer", "trigger": "deadline_expired" } } ``` ```json theme={null} { "decision_id": "dec_uuid", "selected_option_id": "approve", "respondent_note": "Looks good, ship it.", "responded_at": "2026-02-16T14:32:00Z", "response_method": "interactive" } ``` ```json theme={null} { "alert_id": "alt_uuid", "severity": "info", "category": "workflow", "title": "Build pipeline completed", "summary": "All 847 tests passed in 3m 42s. Coverage at 91.2%.", "detail": { "format": "markdown", "content": "## Test Results\n- Unit: 612 passed\n- Integration: 235 passed" }, "actionable": false } ``` ```json theme={null} { "agent_status": "working", "current_task": { "task_id": "tsk_uuid", "description": "Running integration test suite", "progress_pct": 67, "estimated_completion": "2026-02-16T14:45:00Z" }, "resource_usage": { "tokens_consumed_24h": 14500, "api_calls_24h": 230 } } ``` ```json theme={null} { "action_id": "act_uuid", "action_type": "deployment", "title": "Production deployment completed", "status": "success", "summary": "v2.3.1 deployed to 3/3 nodes.", "detail": { "started_at": "2026-02-16T14:33:00Z", "completed_at": "2026-02-16T14:35:42Z", "duration_ms": 162000 }, "triggered_by": { "decision_id": "dec_uuid" }, "rollback_available": true, "before_state": { "version": "2.3.0" }, "after_state": { "version": "2.3.1" } } ``` *** ## Audit Event Schema Every message generates an audit event that maps directly to the [OTel LogRecord](https://opentelemetry.io/docs/specs/otel/logs/data-model/#log-and-event-record-definition) data model. ### Structure ```json theme={null} { "audit_event_id": "evt_uuid_v7", "timestamp": "2026-02-16T14:32:00.123456789Z", "observed_timestamp": "2026-02-16T14:32:00.125000000Z", "trace_id": "a1b2c3d4e5f67890a1b2c3d4e5f67890", "span_id": "1234567890abcdef", "parent_span_id": "abcdef1234567890", "trace_flags": 1, "severity_number": 9, "severity_text": "INFO", "body": { "event_type": "message_delivered", "message_id": "msg_uuid", "message_type": "decision_request", "summary": "Decision request delivered: Deploy v2.3.1 to production?" }, "resource": { "service.name": "agentvault", "service.version": "0.1.0", "av.tenant.id": "tnt_uuid", "av.environment": "production" }, "attributes": { "av.sender.entity_id": "agt_uuid", "av.sender.entity_type": "agent", "av.sender.hub_address": "claude-coder.alice.agentvault.hub", "av.recipient.entity_id": "usr_uuid", "av.recipient.entity_type": "user", "av.conversation.id": "conv_uuid", "av.message.type": "decision_request", "av.message.priority": "high", "av.delivery.latency_ms": 42 }, "hash_chain": { "event_hash": "sha256:a1b2c3d4...", "previous_hash": "sha256:e5f6a7b8...", "sequence_number": 10847 } } ``` ### OTel Field Mapping | OTel LogRecord Field | AgentVault Field | Notes | | -------------------- | -------------------- | -------------------------------------------- | | `Timestamp` | `timestamp` | When the event occurred | | `ObservedTimestamp` | `observed_timestamp` | When the event was recorded (broker latency) | | `TraceId` | `trace_id` | From message envelope | | `SpanId` | `span_id` | From message envelope | | `TraceFlags` | `trace_flags` | `1` = sampled (always for audit) | | `SeverityNumber` | `severity_number` | OTel severity scale (1--24) | | `SeverityText` | `severity_text` | DEBUG/INFO/WARN/ERROR/FATAL | | `Body` | `body` | Event description object | | `Resource` | `resource` | Service + tenant identification | | `Attributes` | `attributes` | Event-specific key-value pairs | ### Severity Mapping | Event Type | OTel Severity | Number | | -------------------- | ------------- | ------ | | `heartbeat` | DEBUG | 5 | | `message_delivered` | INFO | 9 | | `message_read` | INFO | 9 | | `agent_connected` | INFO | 9 | | `agent_disconnected` | INFO | 9 | | `decision_made` | INFO | 10 | | `policy_evaluated` | INFO | 10 | | `action_executed` | INFO | 10 | | `error` | ERROR | 17 | | `security_violation` | FATAL | 21 | *** ## Hash Chain Tamper evidence without blockchain. Each audit event references the hash of the previous event, forming a per-tenant chain. ### Hash Computation The hash covers: previous hash, timestamp, trace context, event body, and participants. It excludes: `observed_timestamp`, delivery metadata, and the `hash_chain` block itself. ```python theme={null} import hashlib import json def compute_event_hash(event: dict, previous_hash: str) -> str: hash_input = { "previous_hash": previous_hash, "timestamp": event["timestamp"], "trace_id": event["trace_id"], "span_id": event["span_id"], "body": event["body"], "sender": event["attributes"].get("av.sender.entity_id"), "recipient": event["attributes"].get("av.recipient.entity_id"), "sequence_number": event["hash_chain"]["sequence_number"] } canonical = json.dumps(hash_input, sort_keys=True, separators=(",", ":")) return "sha256:" + hashlib.sha256(canonical.encode("utf-8")).hexdigest() GENESIS_HASH = "sha256:" + hashlib.sha256(b"agentvault_genesis_v1").hexdigest() ``` ### Chain Verification ```python theme={null} def verify_chain(events: list[dict]) -> tuple[bool, int | None]: """Returns (True, None) if valid or (False, break_index) if tampered.""" for i, event in enumerate(events): expected_previous = GENESIS_HASH if i == 0 else events[i-1]["hash_chain"]["event_hash"] if event["hash_chain"]["previous_hash"] != expected_previous: return (False, i) recomputed = compute_event_hash(event, expected_previous) if event["hash_chain"]["event_hash"] != recomputed: return (False, i) return (True, None) ``` ### Chain Partitioning Chains are partitioned **per tenant** to avoid bottlenecks. Within a tenant, events are strictly ordered by `sequence_number`. Cross-tenant verification is handled via signed checkpoint hashes in the federation phase. ``` Tenant A: genesis -> evt_1 -> evt_2 -> evt_3 -> ... Tenant B: genesis -> evt_1 -> evt_2 -> ... ``` *** ## Storage Schema ### Core Tables ```sql theme={null} CREATE TABLE audit_events ( audit_event_id UUID PRIMARY KEY, tenant_id UUID NOT NULL REFERENCES tenants(tenant_id), sequence_number BIGINT NOT NULL, timestamp TIMESTAMPTZ NOT NULL, observed_timestamp TIMESTAMPTZ NOT NULL DEFAULT now(), trace_id CHAR(32) NOT NULL, span_id CHAR(16) NOT NULL, parent_span_id CHAR(16), trace_flags SMALLINT DEFAULT 1, severity_number SMALLINT NOT NULL, severity_text TEXT NOT NULL, body JSONB NOT NULL, resource JSONB NOT NULL, attributes JSONB NOT NULL, event_hash TEXT NOT NULL, previous_hash TEXT NOT NULL, UNIQUE (tenant_id, sequence_number) ); ``` **Indexes:** * `(tenant_id, sequence_number)` -- Chain traversal * `(trace_id)` -- Conversation/room lookup * `(tenant_id, timestamp)` -- Time-range queries * `(tenant_id, severity_number) WHERE severity_number >= 17` -- Error/critical filtering * GIN on `body` and `attributes` -- JSONB content search ```sql theme={null} CREATE TABLE messages ( message_id UUID PRIMARY KEY, conversation_id UUID NOT NULL REFERENCES conversations(conversation_id), trace_id CHAR(32) NOT NULL, span_id CHAR(16) NOT NULL, parent_span_id CHAR(16), sender_id UUID NOT NULL REFERENCES entities(entity_id), recipient_id UUID NOT NULL REFERENCES entities(entity_id), message_type TEXT NOT NULL, priority TEXT DEFAULT 'normal', payload JSONB NOT NULL, metadata JSONB DEFAULT '{}', envelope_version TEXT DEFAULT '1.0.0', created_at TIMESTAMPTZ DEFAULT now(), delivered_at TIMESTAMPTZ, read_at TIMESTAMPTZ ); ``` ```sql theme={null} CREATE TABLE decisions ( decision_id UUID PRIMARY KEY, conversation_id UUID NOT NULL REFERENCES conversations(conversation_id), request_message_id UUID NOT NULL REFERENCES messages(message_id), response_message_id UUID REFERENCES messages(message_id), status TEXT DEFAULT 'pending' CHECK (status IN ('pending', 'resolved', 'expired', 'auto_resolved')), deadline TIMESTAMPTZ, resolved_at TIMESTAMPTZ, selected_option TEXT, created_at TIMESTAMPTZ DEFAULT now() ); ``` ### Key Query Patterns ```sql theme={null} -- Full audit trail for a conversation SELECT * FROM audit_events WHERE trace_id = :trace_id ORDER BY sequence_number ASC; -- Critical/error events for tenant in last 24h SELECT * FROM audit_events WHERE tenant_id = :tenant_id AND severity_number >= 17 AND timestamp > now() - interval '24 hours' ORDER BY timestamp DESC; -- Hash chain integrity verification SELECT sequence_number, event_hash, previous_hash, LAG(event_hash) OVER (ORDER BY sequence_number) as expected_previous FROM audit_events WHERE tenant_id = :tenant_id ORDER BY sequence_number; ``` *** ## API Endpoints ### Audit Trail ``` GET /v1/audit/trace/{trace_id} GET /v1/audit/tenant?since=...&until=...&severity_min=17 GET /v1/audit/entity/{entity_id}?limit=100 ``` ### Chain Verification ```bash theme={null} POST /v1/audit/verify ``` ```json theme={null} { "tenant_id": "tnt_uuid", "from_sequence": 1, "to_sequence": 10847 } ``` Response: ```json theme={null} { "valid": true, "events_verified": 10847, "first_hash": "sha256:...", "last_hash": "sha256:...", "verified_at": "2026-02-16T15:00:00Z" } ``` ### Pending Decisions ``` GET /v1/decisions/pending?user_id={user_id} ``` *** ## Phase Evolution The data model is designed to grow without schema changes across five phases. * `trace_id` = conversation\_id (1:1 mapping) * `span_id` = each message exchange * Single chain per tenant * Postgres storage, direct queries * No external OTel infrastructure needed * `trace_id` = room\_id * Multiple agents produce spans within the same trace * `parent_span_id` links replies to triggers * New event types: `agent_joined_room`, `agent_left_room` * Trace context propagates across agent boundaries * OTel SpanLinks for delegation chains referencing original traces * Policy evaluation events added to audit log * OTel Collector deployed as export sidecar * Resource attributes expand: `av.team.id`, `av.workspace.id` * RBAC audit events: `permission_granted`, `permission_denied`, `role_changed` * OTel export to customer backends becomes a selling point * Cross-tenant trace context propagation (W3C Trace Context headers) * Signed checkpoint hashes between federated tenants * Full OTel Collector deployment with multi-backend export *** ## OTel Export Path When ready, adding OTel export requires zero schema changes because the data model is already OTel-compatible: ```python Phase 1 (Postgres only) theme={null} async def emit_audit_event(event: AuditEvent): await db.insert_audit_event(event) ``` ```python Phase 3+ (Add OTel export) theme={null} async def emit_audit_event(event: AuditEvent): await db.insert_audit_event(event) # Still Postgres otel_record = event.to_otel_log_record() # Trivial -- schema already matches logger.emit(otel_record) ``` OTel Collector configuration for customer export: ```yaml theme={null} # otel-collector-config.yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 processors: batch: timeout: 5s filter: logs: include: match_type: strict resource_attributes: - key: av.tenant.id value: "tnt_enterprise_customer" exporters: splunkhec: endpoint: "https://customer-splunk.example.com:8088" datadog: api: key: "${DD_API_KEY}" service: pipelines: logs: receivers: [otlp] processors: [batch, filter] exporters: [splunkhec, datadog] ``` Compatible with OpenTelemetry Specification v1.29+. *** ## Telemetry Span Types AgentVault emits **21 span types** using the `av.` prefix naming convention. These spans are generated by the plugin, client SDK, and backend. ### Core Agent Spans | Span Builder | Span Name | Description | | ----------------- | ----------- | ------------------------------------------------ | | `buildLlmSpan` | `av.llm` | LLM inference (model, tokens, latency, provider) | | `buildToolSpan` | `av.tool` | Tool execution (name, success/error, latency) | | `buildErrorSpan` | `av.error` | Error events (type, message, kind) | | `buildHttpSpan` | `av.http` | HTTP client calls (method, URL, status) | | `buildActionSpan` | `av.action` | Agent action execution (type, target) | | `buildNavSpan` | `av.nav` | Navigation/URL changes | | `buildTaskSpan` | `av.task` | Task execution (status, steps, tokens) | | `buildEvalSpan` | `av.eval` | Evaluation/scoring (name, score, explanation) | ### Platform Spans | Span Builder | Span Name | Description | | -------------------------- | -------------------- | ------------------------------------------------------ | | `buildSkillInvocationSpan` | `av.skill.invoke` | Skill invocation lifecycle (phase, version, cost) | | `buildForgeSpan` | `av.forge` | Skill Builder phases (create/brainstorm/build/publish) | | `buildPolicyViolationSpan` | `av.policy.evaluate` | Policy enforcement (rule, scope, action) | | `buildDecisionSpan` | `av.decision` | Decision request/response lifecycle | | `buildCapabilitySpan` | `av.capability` | Skill registration and capability changes | ### Infrastructure Spans | Span Builder | Span Name | Description | | --------------------- | --------------- | ------------------------------------------------------------------------------------------------------ | | `buildResyncSpan` | `av.resync` | Double Ratchet resynchronization events | | `buildA2aSpan` | `av.a2a` | Agent-to-agent channel operations | | `buildScanSpan` | `av.scan` | Policy/content scanning results | | `buildWorkspaceSpan` | `av.workspace` | Workspace file operations (upload/download/scan) | | `buildEnrollmentSpan` | `av.enrollment` | Device enrollment events | | `buildRoomSpan` | `av.room` | Room lifecycle (create/join/rekey/message) | | `buildTrustSpan` | `av.trust` | Trust score changes and tier transitions | | `buildEvalRunSpan` | `av.eval_run` | Eval framework test execution (scenario\_id, variation\_id, shift\_magnitude, match\_outcome, flagged) | ### W3C Trace Context All spans include W3C TraceContext headers for cross-agent trace correlation: ```typescript theme={null} import { buildTraceparent, propagateContext } from "@agentvault/crypto"; // Create a traceparent header const header = buildTraceparent(traceId, spanId); // "00-{traceId}-{spanId}-01" // Propagate context to a child span const child = propagateContext(parentTraceparent); ```