> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agentvault.chat/llms.txt
> Use this file to discover all available pages before exploring further.

# Messaging Reliability

> How AgentVault ensures reliable message delivery across network interruptions.

AgentVault's messaging reliability system is inspired by Signal's delivery architecture.
It ensures that encrypted messages reach their recipients even when WebSocket connections
die silently, devices go offline, or networks switch mid-conversation.

## The Problem

When a client's WebSocket connection dies silently (iOS backgrounding, idle desktop, network
switch), the backend may still consider the device "connected." Without reliability measures,
messages are stored but never delivered -- the server believes the device received them,
and push notifications are suppressed.

## Architecture Overview

```
Owner App                    Backend                        Agent Plugin
    |                           |                               |
    |--- WS message ----------->|                               |
    |                           |-- store ciphertext            |
    |                           |-- create delivery record      |
    |                           |-- publish to Redis            |
    |                           |-- direct WS forward --------->|
    |                           |-- dispatch push (always-on)   |
    |                           |                               |
    |                           |<--- ACK ----------------------|
    |                           |-- mark DELIVERED              |
    |                           |                               |
    |                   [60s escalation loop]                    |
    |                           |-- query stale PENDING rows    |
    |                           |-- re-trigger push             |
    |                           |-- mark FAILED after 3 retries |
```

The reliability system has four interlocking phases: push always-on, delivery receipts,
connection liveness detection, and push notification navigation.

## Phase 1: Push Always-On

<Note>
  Push notifications fire on **every message** regardless of WebSocket connection state.
  A redundant push notification is harmless; a missed one is a delivery failure.
</Note>

Traditional systems suppress push notifications when the backend believes a device is connected
via WebSocket. This breaks when the WebSocket dies silently (no close frame is sent). AgentVault
removes this suppression entirely.

Push notifications are safe to send unconditionally because:

* They show an OS-level alert (service worker on web, Expo Push on native)
* They do **not** inject messages into chat state -- that comes from WebSocket or history sync
* Duplicate alerts are a minor UX inconvenience; missed messages are a reliability failure

## Phase 2: Delivery Receipts (ACK System)

Every message creates a delivery tracking record for each recipient device. The ACK system
closes the loop on whether a message was actually received and decrypted.

### Delivery Flow

<Steps>
  <Step title="Message sent">
    After storing ciphertext in the `messages` table, the backend creates a `MessageDelivery`
    row for each recipient device (excluding the sender) with status `PENDING`.
  </Step>

  <Step title="Client receives and decrypts">
    After successful decryption, the client sends an ACK event over WebSocket.
  </Step>

  <Step title="ACK processed">
    The backend marks the delivery as `DELIVERED` with a timestamp.
  </Step>

  <Step title="Escalation (if no ACK)">
    A background task checks for stale `PENDING` rows every 60 seconds
    and re-triggers delivery attempts.
  </Step>
</Steps>

### ACK Wire Protocol

After decrypting one or more messages, the client sends a batch ACK:

```json theme={null}
{
  "event": "ack",
  "data": {
    "message_ids": ["uuid-1", "uuid-2", "uuid-3"]
  }
}
```

| Behavior          | Value                                   |
| ----------------- | --------------------------------------- |
| Max IDs per ACK   | 50                                      |
| Client debounce   | 500ms                                   |
| Server processing | Fire-and-forget (`asyncio.create_task`) |

<Tip>
  Both the web app and the `@agentvault/agentvault` plugin implement ACK batching and debouncing
  automatically. You do not need to manage this manually unless building a custom client.
</Tip>

### Delivery Statuses

```sql theme={null}
CREATE TYPE delivery_status AS ENUM (
  'pending',
  'delivered',
  'read',
  'failed'
);
```

| Status      | Meaning                                             |
| ----------- | --------------------------------------------------- |
| `pending`   | Message stored, delivery not yet confirmed          |
| `delivered` | Client sent ACK after successful decryption         |
| `read`      | Client confirmed the message was displayed (future) |
| `failed`    | Delivery failed after 3 escalation attempts         |

### Escalation Task

A background task runs every 60 seconds to catch undelivered messages.

| Parameter       | Value | Description                                          |
| --------------- | ----- | ---------------------------------------------------- |
| Interval        | 60s   | How often the escalation loop runs                   |
| Stale threshold | 120s  | Deliveries older than 2 minutes are considered stale |
| Max per cycle   | 100   | Bounds work per loop iteration                       |
| Max escalations | 3     | Retries before marking as `FAILED`                   |

<Warning>
  Without a retry cap, stale deliveries cause infinite escalation loops that hammer
  failed endpoints every 60 seconds. The 3-attempt cap prevents this.
</Warning>

The escalation task:

1. Queries deliveries where `status = 'pending'`, `created_at < 2 minutes ago`, and `escalation_count < 3`
2. Re-triggers push notification for owner devices
3. Increments `escalation_count` on each attempt
4. Marks delivery as `FAILED` after 3 unsuccessful escalations

## Phase 3: Connection Liveness

AgentVault detects zombie WebSocket connections through a two-sided heartbeat system.

### Server-Side: Pong Timeout + Redis TTL

| Constant             | Value | Purpose                                 |
| -------------------- | ----- | --------------------------------------- |
| `HEARTBEAT_INTERVAL` | 30s   | How often server sends ping             |
| `HEARTBEAT_TIMEOUT`  | 90s   | Max time without pong before disconnect |
| `REDIS_ALIVE_TTL`    | 60s   | Redis key auto-expiry                   |

The server tracks the last pong timestamp for each device. If no pong is received within 90
seconds, the connection is forcibly closed and the device is removed from the active connections
map.

Redis liveness keys provide crash resilience:

```
device:{uuid}:alive  ->  "1"  TTL=60s
```

* **SET** on connect, **EXPIRE** refreshed on each pong, **DEL** on disconnect
* Auto-expires if the backend process crashes (no explicit cleanup needed)

### Client-Side: Ping Watchdog

Both the web app and plugin run a 45-second watchdog timer (1.5x the server's ping interval).

<CodeGroup>
  ```typescript Web App theme={null}
  // Reset watchdog on every server ping
  const WATCHDOG_TIMEOUT = 45_000; // 1.5x server interval

  let watchdog: NodeJS.Timeout;

  ws.onmessage = (event) => {
    clearTimeout(watchdog);
    watchdog = setTimeout(() => {
      ws.close(4000, "Ping timeout");
      // Reconnect with exponential backoff
    }, WATCHDOG_TIMEOUT);
  };
  ```

  ```typescript Plugin theme={null}
  // Equivalent watchdog in the agent plugin
  private resetPingWatchdog() {
    clearTimeout(this.pingWatchdog);
    this.pingWatchdog = setTimeout(() => {
      this.ws?.close(4000, "Ping timeout");
      this.scheduleReconnect();
    }, 45_000);
  }
  ```
</CodeGroup>

When the watchdog fires, the client closes the WebSocket with code `4000` and triggers
the standard reconnect flow with exponential backoff.

## Phase 4: Push Notification Handling

### iOS and Android

Push notifications carry the `conversation_id` in their data payload. Tapping a notification
routes the user directly to the relevant chat.

* **Cold start:** `Notifications.getLastNotificationResponseAsync()` checks for a pending notification on app launch
* **Background:** `Notifications.addNotificationResponseReceivedListener` handles taps while the app is backgrounded

### Foreground Display

When the app is in the foreground, push notifications are still displayed as alerts:

```typescript theme={null}
Notifications.setNotificationHandler({
  handleNotification: async () => ({
    shouldShowAlert: true,
    shouldPlaySound: true,
    shouldSetBadge: true,
  }),
});
```

### Push Payload

Each push notification includes enriched metadata for reliable delivery tracking:

| Field        | Value        | Purpose                       |
| ------------ | ------------ | ----------------------------- |
| `sound`      | `"default"`  | Audible alert                 |
| `priority`   | `"high"`     | Bypass battery optimization   |
| `channelId`  | `"messages"` | Android notification channel  |
| `message_id` | UUID         | Delivery tracking correlation |

## Agent Delivery Model

Agents connect via the `@agentvault/agentvault` npm plugin over WebSocket. There is no
webhook delivery -- agent gateways typically run locally without a public URL.

<Steps>
  <Step title="Real-time delivery">
    The plugin maintains a persistent WebSocket connection. The server forwards ciphertext
    directly over the socket.
  </Step>

  <Step title="History sync on reconnect">
    On reconnect, the plugin fetches missed messages via the history endpoint:

    ```
    GET /api/v1/devices/{id}/messages?since={last_seen_timestamp}
    ```
  </Step>

  <Step title="ACK after decryption">
    After decrypting each message, the plugin batches ACKs and sends them back to the server.
  </Step>
</Steps>

<Note>
  After updating the `@agentvault/agentvault` npm package, the OpenClaw gateway must be
  restarted (`openclaw gateway restart`) for new code to take effect. Node.js caches modules
  in memory at import time.
</Note>

## Reconnection Strategy

Both the web app and plugin use exponential backoff with jitter for reconnection.

| Attempt | Base Delay | Max Delay |
| ------- | ---------- | --------- |
| 1       | 1s         | --        |
| 2       | 2s         | --        |
| 3       | 4s         | --        |
| 4       | 8s         | --        |
| 5+      | 16s        | 30s       |

Additionally, the web app listens for `visibilitychange` events to detect when a browser tab
returns to the foreground and immediately attempts reconnection, bypassing the backoff timer.

<Accordion title="Wake Detection">
  The plugin includes a wake detector that monitors gaps in the event loop. If no events
  are processed for more than 120 seconds (indicating the host machine was asleep or the
  LLM was running a long inference), the plugin immediately reconnects and performs a
  history sync to catch any missed messages.

  The 120-second threshold is intentionally generous -- LLM inference can block the Node.js
  event loop for 30-60 seconds, and a shorter threshold would trigger false wake detections.
</Accordion>

## Design Principles

These principles guided the reliability system's design:

1. **Never suppress push based on WS state.** Silent disconnections are common on mobile. Push is cheap; missed messages are not.
2. **ACK means decrypted, not received.** The delivery receipt is sent only after the client successfully decrypts the ciphertext, not merely when the WebSocket frame arrives.
3. **Retry caps are essential.** Without them, stale deliveries cause infinite escalation loops.
4. **Redundancy over elegance.** It is better to deliver a message twice than to miss it entirely.