Skip to main content

​
Shadow Mode

Shadow Mode implements a progressive autonomy model for AI agents. Instead of granting full autonomy on day one, agents earn independence by demonstrating alignment with human decisions over time. The model has three levels:
LevelBehaviorHuman Role
ShadowAgent observes and records what it would recommend, but takes no actionHuman decides independently
SupervisedAgent recommends an action, human confirms or overrides before executionHuman approves or rejects
AutonomousAgent acts independently, human reviews after the factHuman monitors
Every agent starts at Shadow. Graduation to higher levels requires meeting quantitative gates — not just vibes.

​
Creating Shadow Sessions

A shadow session tracks one specific capability being evaluated. You define the skill (what the agent can do) and the decision class (what type of decision it involves). 
POST /api/v1/shadow/sessions
Authorization: Bearer <jwt>
Content-Type: application/json

{
  "device_id": "3a7df4c2-...",
  "skill_name": "incident_response",
  "decision_class": "escalation",
  "config": {
    "min_observations": 25,
    "agreement_threshold": 0.90,
    "critical_miss_window_hours": 168
  }
}
FieldDescription
skill_nameThe agent capability being evaluated (e.g., incident_response, code_review, deployment)
decision_classThe type of decision (e.g., escalation, approval, rollback, triage)
min_observationsNumber of decisions required before graduation is possible
agreement_thresholdMinimum agreement rate (0.0 - 1.0) required to graduate
critical_miss_window_hoursTime window in which no critical misses can occur
A session is scoped to one skill + decision class combination. An agent can have multiple active sessions — for example, incident_response/escalation at Supervised while deployment/approval is still in Shadow.

​
Agreement Tracking

During a shadow session, every decision point follows the same flow:
1

Agent recommends

The agent analyzes the situation and records what it would recommend. In Shadow mode, this recommendation is silent — the human never sees it. In Supervised mode, the recommendation is shown to the human.
2

Human decides

The human makes the actual decision independently (Shadow) or in response to the agent’s recommendation (Supervised).
3

System compares

The platform compares the agent’s recommendation against the human’s decision and records the outcome: agree or disagree.
POST /api/v1/shadow/sessions/{session_id}/record-human-decision
Authorization: Bearer <jwt>
Content-Type: application/json

{
  "observation_id": "obs_uuid",
  "agent_recommendation": {
    "action": "escalate",
    "confidence": 0.87,
    "reasoning": "Production error rate exceeded 5% threshold"
  },
  "human_decision": {
    "action": "escalate",
    "notes": "Agreed — error rate is climbing fast"
  }
}
The agreement rate is computed as a rolling metric: 
agreement_rate = agreed_decisions / total_decisions
The platform tracks this continuously and displays it on the session dashboard alongside a trend line showing agreement over time.
Disagreements are not failures — they are learning signals. Review disagreements to understand whether the agent needs tuning or whether the human’s decision was an edge case.

​
Graduation Gates

Graduation from one level to the next requires meeting three quantitative requirements simultaneously:
GateRequirementExample
Minimum observationsEnough decisions have been recorded to be statistically meaningful25 observations
Agreement thresholdAgreement rate meets or exceeds the configured threshold90% agreement
No recent critical missesNo critical disagreements within the lookback windowZero critical misses in last 7 days
A critical miss is a disagreement where the agent’s recommendation, if followed, would have caused significant harm — as determined by the human reviewer when recording the decision.

​
Checking Eligibility

GET /api/v1/shadow/sessions/{session_id}/graduation-check
Authorization: Bearer <jwt>
Response: 
{
  "eligible": true,
  "current_level": "shadow",
  "next_level": "supervised",
  "gates": {
    "min_observations": { "required": 25, "actual": 31, "met": true },
    "agreement_threshold": { "required": 0.90, "actual": 0.94, "met": true },
    "critical_miss_window": { "window_hours": 168, "misses": 0, "met": true }
  }
}

​
Promoting

Graduation is an explicit action — it never happens automatically. When all gates are met, the owner promotes the session: 
POST /api/v1/shadow/sessions/{session_id}/graduate
Authorization: Bearer <jwt>
This advances the session from Shadow to Supervised, or from Supervised to Autonomous.
Graduation can be reversed. If an agent at Supervised or Autonomous level has a critical miss, the owner can demote it back to a lower level at any time.

​
When to Use Shadow Mode

Shadow Mode is most valuable in four situations:

​
New Skills

When an agent gains a new capability (e.g., you add a deployment tool), start that skill in Shadow. Let the agent build a track record before it acts independently.

​
High-Risk Decision Classes

For decisions with significant consequences — production deployments, access grants, financial transactions — Shadow Mode provides a safety net even for well-established agents.

​
Post-Incident Recovery

After an agent makes a mistake, demote the relevant skill back to Shadow. Rebuild confidence through measured agreement before restoring autonomy.

​
New Agent Versions

When you update the underlying model or significantly change the agent’s prompts, re-enter Shadow Mode for critical skills. Model updates can change behavior in unexpected ways.

​
Integration

The agent plugin reports recommendations and receives human decisions through the shadow session API.

​
Plugin-Side Flow

  1. Agent encounters a decision point during normal operation
  2. Agent computes its recommendation and submits it to the session
  3. If in Shadow mode: recommendation is recorded silently, human decides independently
  4. If in Supervised mode: recommendation is shown to the human in the AgentVault dashboard
  5. Human decision is recorded via the API
  6. System compares and updates the agreement rate

​
Recording from the Plugin

// Agent-side: record what the agent would recommend
await fetch(`${API_BASE}/api/v1/shadow/sessions/${sessionId}/record-human-decision`, {
  method: "POST",
  headers: { Authorization: `Bearer ${jwt}` },
  body: JSON.stringify({
    observation_id: observationId,
    agent_recommendation: {
      action: "approve_deploy",
      confidence: 0.92,
      reasoning: "All tests passing, no breaking changes detected",
    },
    human_decision: {
      action: humanAction,
      notes: humanNotes,
    },
  }),
});
The dashboard displays a live view of the session: observation count, agreement rate, trend line, recent disagreements, and graduation eligibility.

​
API Reference

MethodEndpointDescription
POST/api/v1/shadow/sessionsCreate a new shadow session
POST/api/v1/shadow/sessions/{id}/record-human-decisionRecord an agent recommendation and human decision
GET/api/v1/shadow/sessions/{id}/graduation-checkCheck graduation eligibility against all gates
POST/api/v1/shadow/sessions/{id}/graduatePromote session to the next autonomy level