Shadow Mode

Shadow Mode implements a progressive autonomy model for AI agents. Instead of granting full autonomy on day one, agents earn independence by demonstrating alignment with human decisions over time. The model has three levels:

Level	Behavior	Human Role
Shadow	Agent observes and records what it would recommend, but takes no action	Human decides independently
Supervised	Agent recommends an action, human confirms or overrides before execution	Human approves or rejects
Autonomous	Agent acts independently, human reviews after the fact	Human monitors

Every agent starts at Shadow. Graduation to higher levels requires meeting quantitative gates — not just vibes.

Creating Shadow Sessions

A shadow session tracks one specific capability being evaluated. You define the skill (what the agent can do) and the decision class (what type of decision it involves).

POST /api/v1/shadow/sessions
Authorization: Bearer <jwt>
Content-Type: application/json

{
  "device_id": "3a7df4c2-...",
  "skill_name": "incident_response",
  "decision_class": "escalation",
  "config": {
    "min_observations": 25,
    "agreement_threshold": 0.90,
    "critical_miss_window_hours": 168
  }
}

Field	Description
`skill_name`	The agent capability being evaluated (e.g., `incident_response`, `code_review`, `deployment`)
`decision_class`	The type of decision (e.g., `escalation`, `approval`, `rollback`, `triage`)
`min_observations`	Number of decisions required before graduation is possible
`agreement_threshold`	Minimum agreement rate (0.0 - 1.0) required to graduate
`critical_miss_window_hours`	Time window in which no critical misses can occur

A session is scoped to one skill + decision class combination. An agent can have multiple active sessions — for example, incident_response/escalation at Supervised while deployment/approval is still in Shadow.

Agreement Tracking

During a shadow session, every decision point follows the same flow:

Agent recommends

The agent analyzes the situation and records what it would recommend. In Shadow mode, this recommendation is silent — the human never sees it. In Supervised mode, the recommendation is shown to the human.

Human decides

The human makes the actual decision independently (Shadow) or in response to the agent’s recommendation (Supervised).

System compares

The platform compares the agent’s recommendation against the human’s decision and records the outcome: agree or disagree.

POST /api/v1/shadow/sessions/{session_id}/record-human-decision
Authorization: Bearer <jwt>
Content-Type: application/json

{
  "observation_id": "obs_uuid",
  "agent_recommendation": {
    "action": "escalate",
    "confidence": 0.87,
    "reasoning": "Production error rate exceeded 5% threshold"
  },
  "human_decision": {
    "action": "escalate",
    "notes": "Agreed — error rate is climbing fast"
  }
}

The agreement rate is computed as a rolling metric:

agreement_rate = agreed_decisions / total_decisions

The platform tracks this continuously and displays it on the session dashboard alongside a trend line showing agreement over time.

Disagreements are not failures — they are learning signals. Review disagreements to understand whether the agent needs tuning or whether the human’s decision was an edge case.

Graduation Gates

Graduation from one level to the next requires meeting three quantitative requirements simultaneously:

Gate	Requirement	Example
Minimum observations	Enough decisions have been recorded to be statistically meaningful	25 observations
Agreement threshold	Agreement rate meets or exceeds the configured threshold	90% agreement
No recent critical misses	No critical disagreements within the lookback window	Zero critical misses in last 7 days

A critical miss is a disagreement where the agent’s recommendation, if followed, would have caused significant harm — as determined by the human reviewer when recording the decision.

Checking Eligibility

GET /api/v1/shadow/sessions/{session_id}/graduation-check
Authorization: Bearer <jwt>

Response:

{
  "eligible": true,
  "current_level": "shadow",
  "next_level": "supervised",
  "gates": {
    "min_observations": { "required": 25, "actual": 31, "met": true },
    "agreement_threshold": { "required": 0.90, "actual": 0.94, "met": true },
    "critical_miss_window": { "window_hours": 168, "misses": 0, "met": true }
  }
}

Promoting

Graduation is an explicit action — it never happens automatically. When all gates are met, the owner promotes the session:

POST /api/v1/shadow/sessions/{session_id}/graduate
Authorization: Bearer <jwt>

This advances the session from Shadow to Supervised, or from Supervised to Autonomous.

Graduation can be reversed. If an agent at Supervised or Autonomous level has a critical miss, the owner can demote it back to a lower level at any time.

When to Use Shadow Mode

Shadow Mode is most valuable in four situations:

New Skills

When an agent gains a new capability (e.g., you add a deployment tool), start that skill in Shadow. Let the agent build a track record before it acts independently.

High-Risk Decision Classes

For decisions with significant consequences — production deployments, access grants, financial transactions — Shadow Mode provides a safety net even for well-established agents.

Post-Incident Recovery

After an agent makes a mistake, demote the relevant skill back to Shadow. Rebuild confidence through measured agreement before restoring autonomy.

New Agent Versions

When you update the underlying model or significantly change the agent’s prompts, re-enter Shadow Mode for critical skills. Model updates can change behavior in unexpected ways.

Integration

The agent plugin reports recommendations and receives human decisions through the shadow session API.

Plugin-Side Flow

Agent encounters a decision point during normal operation
Agent computes its recommendation and submits it to the session
If in Shadow mode: recommendation is recorded silently, human decides independently
If in Supervised mode: recommendation is shown to the human in the AgentVault dashboard
Human decision is recorded via the API
System compares and updates the agreement rate

Recording from the Plugin

// Agent-side: record what the agent would recommend
await fetch(`${API_BASE}/api/v1/shadow/sessions/${sessionId}/record-human-decision`, {
  method: "POST",
  headers: { Authorization: `Bearer ${jwt}` },
  body: JSON.stringify({
    observation_id: observationId,
    agent_recommendation: {
      action: "approve_deploy",
      confidence: 0.92,
      reasoning: "All tests passing, no breaking changes detected",
    },
    human_decision: {
      action: humanAction,
      notes: humanNotes,
    },
  }),
});

The dashboard displays a live view of the session: observation count, agreement rate, trend line, recent disagreements, and graduation eligibility.

API Reference

Method	Endpoint	Description
`POST`	`/api/v1/shadow/sessions`	Create a new shadow session
`POST`	`/api/v1/shadow/sessions/{id}/record-human-decision`	Record an agent recommendation and human decision
`GET`	`/api/v1/shadow/sessions/{id}/graduation-check`	Check graduation eligibility against all gates
`POST`	`/api/v1/shadow/sessions/{id}/graduate`	Promote session to the next autonomy level

Introduction & Core Concepts

Agent Identity & Cryptography

Secure Communications

Trust Scoring & Authorization

Agent Builder

Quality Control

Integrations

The AI Skill Marketplace

Observability & Audit Trails

Security & Compliance

API Reference

Shadow Mode

Shadow Mode

Creating Shadow Sessions

Agreement Tracking

Graduation Gates

Checking Eligibility

Promoting

When to Use Shadow Mode

New Skills

High-Risk Decision Classes

Post-Incident Recovery

New Agent Versions

Integration

Plugin-Side Flow

Recording from the Plugin

API Reference

​Shadow Mode

​Creating Shadow Sessions

​Agreement Tracking

​Graduation Gates

​Checking Eligibility

​Promoting

​When to Use Shadow Mode

​New Skills

​High-Risk Decision Classes

​Post-Incident Recovery

​New Agent Versions

​Integration

​Plugin-Side Flow

​Recording from the Plugin

​API Reference

Shadow Mode

Creating Shadow Sessions

Agreement Tracking

Graduation Gates

Checking Eligibility

Promoting

When to Use Shadow Mode

New Skills

High-Risk Decision Classes

Post-Incident Recovery

New Agent Versions

Integration

Plugin-Side Flow

Recording from the Plugin

API Reference