Annotation Format Schema v1.0

A portable format for structured UI feedback

Overview

The Annotation Format Schema (AFS) is an open format created and used by Agentation for capturing UI feedback in a way that AI coding agents can reliably parse and act on. Think of it like smart Figma comments for your running app — persistent annotations attached to specific elements, with threads, status tracking, resolution workflows, and structured metadata that agents can actually understand.

This spec defines the annotation object shape. Tools can emit annotations in this format, and agents can consume them regardless of how they were created.

Human
Reviewer
Annotation
Object
Agent
Claude
create→ pendingget_pendingannotation dataacknowledge→ acknowledgedresolve→ resolvedstatus update
→pending
→acknowledged
→resolved

What This Unlocks

A structured schema isn't just about clean data — it enables entirely new workflows:

  • Two-way communication — Agents can reply to annotations, asking “Should this be 24px or 16px?” and get responses in the same thread
  • Status tracking — See what's pending, acknowledged, resolved, or dismissed at a glance
  • Cross-page queries — “What annotations do I have?” works across your entire site
  • Bulk operations — “Clear all annotations” or “Show me blocking issues only”
  • Persistent history — Feedback survives page refreshes and browser sessions

Without a schema, feedback is fire-and-forget. With one, it becomes a conversation.

Design Goals

  • Agent-readable — Structured data that LLMs can parse without guessing
  • Framework-agnostic — Works with any UI, though React gets extra context
  • Tool-agnostic — Any tool can emit, any agent can consume
  • Human-authored — Designed for feedback from humans (or automated reviewers)
  • Minimal core — Few required fields, many optional for richer context

Annotation Object

An annotation represents a single piece of feedback attached to a UI element.

Note: The server may add metadata fields (sessionId, createdAt, updatedAt) when syncing annotations.

Required Fields

{
  id: string;           // Unique identifier (e.g. "ann_abc123")
  comment: string;      // Human feedback ("Button is misaligned")
  elementPath: string;  // CSS selector path ("body > main > button.cta")
  timestamp: number;    // Unix timestamp (ms)
  x: number;            // % of viewport width (0-100)
  y: number;            // px from document top (or viewport if isFixed)
  element: string;      // Tag name ("button", "div", "input")
}

Recommended Fields

{
  url: string;          // Page URL where annotation was created
  boundingBox: {        // Element position at annotation time
    x: number;
    y: number;
    width: number;
    height: number;
  };
}

Optional Context Fields

{
  // React-specific (when available)
  reactComponents: string;  // Component tree ("App > Dashboard > Button")


  // Element details
  cssClasses: string;       // Class list ("btn btn-primary disabled")
  computedStyles: string;   // Key CSS properties
  accessibility: string;    // ARIA attributes, role
  nearbyText: string;       // Visible text in/around element
  selectedText: string;     // Text highlighted by user


  // Feedback classification
  intent: "fix" | "change" | "question" | "approve";
  severity: "blocking" | "important" | "suggestion";
}

Lifecycle Fields

{
  status: "pending" | "acknowledged" | "resolved" | "dismissed";
  resolvedAt: string;       // ISO timestamp
  resolvedBy: "human" | "agent";
  thread: ThreadMessage[];  // Back-and-forth conversation
}

Browser Component Fields

These optional fields are set by the Agentation browser component for UI rendering:

{
  isFixed: boolean;         // Element has fixed/sticky positioning
  isMultiSelect: boolean;   // Created via drag selection
  fullPath: string;         // Full DOM path (vs shorter elementPath)
  nearbyElements: string;   // Info about nearby DOM elements
}

Full TypeScript Definition

type Annotation = {
  // Required
  id: string;
  comment: string;
  elementPath: string;
  timestamp: number;
  x: number;            // % of viewport width (0-100)
  y: number;            // px from document top (or viewport if isFixed)
  element: string;      // Tag name ("button", "div")


  // Recommended
  url?: string;
  boundingBox?: {
    x: number;
    y: number;
    width: number;
    height: number;
  };


  // Optional context
  reactComponents?: string;
  cssClasses?: string;
  computedStyles?: string;
  accessibility?: string;
  nearbyText?: string;
  selectedText?: string;


  // Browser component fields
  isFixed?: boolean;       // Element has fixed/sticky positioning
  isMultiSelect?: boolean; // Created via drag selection
  fullPath?: string;       // Full DOM path
  nearbyElements?: string; // Info about nearby elements


  // Feedback classification
  intent?: "fix" | "change" | "question" | "approve";
  severity?: "blocking" | "important" | "suggestion";


  // Lifecycle
  status?: "pending" | "acknowledged" | "resolved" | "dismissed";
  resolvedAt?: string;
  resolvedBy?: "human" | "agent";
  thread?: ThreadMessage[];
};


type ThreadMessage = {
  id: string;
  role: "human" | "agent";
  content: string;
  timestamp: number;
};

Event Envelope

For real-time streaming, annotations are wrapped in an event envelope:

type AgentationEvent = {
  type: "annotation.created" | "annotation.updated" | "annotation.deleted"
      | "session.created" | "session.updated" | "session.closed"
      | "thread.message" | "action.requested";
  timestamp: string;     // ISO 8601
  sessionId: string;
  sequence: number;      // Monotonic for ordering/replay
  payload: Annotation | Session | ThreadMessage | ActionRequest;
};

The sequence number enables clients to detect missed events and request replay. See MCP for SSE streaming details.

JSON Schema

For validation in any language:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://agentation.dev/schema/annotation.v1.json",
  "title": "Annotation",
  "type": "object",
  "required": ["id", "comment", "elementPath", "timestamp", "x", "y", "element"],
  "properties": {
    "id": { "type": "string" },
    "comment": { "type": "string" },
    "elementPath": { "type": "string" },
    "timestamp": { "type": "number" },
    "x": { "type": "number", "description": "% of viewport width (0-100)" },
    "y": { "type": "number", "description": "px from document top" },
    "element": { "type": "string" },
    "url": { "type": "string", "format": "uri" },
    "boundingBox": {
      "type": "object",
      "properties": {
        "x": { "type": "number" },
        "y": { "type": "number" },
        "width": { "type": "number" },
        "height": { "type": "number" }
      },
      "required": ["x", "y", "width", "height"]
    },
    "reactComponents": { "type": "string" },
    "isFixed": { "type": "boolean" },
    "isMultiSelect": { "type": "boolean" },
    "fullPath": { "type": "string" },
    "nearbyElements": { "type": "string" },
    "intent": { "enum": ["fix", "change", "question", "approve"] },
    "severity": { "enum": ["blocking", "important", "suggestion"] },
    "status": { "enum": ["pending", "acknowledged", "resolved", "dismissed"] }
  }
}

Example Annotation

{
  "id": "ann_k8x2m",
  "comment": "Button is cut off on mobile viewport",
  "elementPath": "body > main > .hero-section > button.cta",
  "timestamp": 1705694400000,
  "x": 45.5,
  "y": 480,
  "element": "button",
  "url": "http://localhost:3000/landing",
  "boundingBox": { "x": 120, "y": 480, "width": 200, "height": 48 },
  "reactComponents": "App > LandingPage > HeroSection > CTAButton",
  "cssClasses": "cta btn-primary",
  "nearbyText": "Get Started Free",
  "intent": "fix",
  "severity": "blocking",
  "status": "pending"
}

Markdown Output Format

For pasting into chat-based agents, annotations can be serialized as markdown:

## Annotation #1
**Element:** button.cta
**Path:** body > main > .hero-section > button.cta
**React:** App > LandingPage > HeroSection > CTAButton
**Position:** 120px, 480px (200×48px)
**Feedback:** Button is cut off on mobile viewport
**Severity:** blocking

See Output Formats for detail level options (Compact → Forensic).

Implementations

Tools that emit or consume this format:

Agentation (React)Click-to-annotate toolbar for React apps
Agentation MCP ServerExposes annotations to Claude Code and other MCP clients

Building an Implementation

To emit Agentation Format annotations from your tool:

  1. Capture the required fields: id, comment, elementPath, timestamp, x, y, element
  2. Add recommended fields for better agent accuracy: url, boundingBox
  3. For React apps, traverse the fiber tree to get reactComponents
  4. Output as JSON for MCP/API consumption, or markdown for chat pasting

See the Agentation source for reference implementations of element detection and React component traversal.

Why This Format?

Existing agent protocols (MCP, A2A, ACP) standardize tools and messaging, but they don't define a UI feedback grammar. They rely on whatever structured context you feed them.

This format fills that gap: a portable wire format specifically for "human points at UI, agent needs to find and fix the code." We hope it's useful to others building similar tools.

Versioning

Current version: v1

Schema URL: https://agentation.dev/schema/annotation.v1.json