Browser Control - SimpleClaw

SimpleClaw’s Browser Control feature enables AI agents to interact with web pages through an integrated Playwright-based automation server.

Overview

Browser control provides:

Multi-profile management - Run multiple isolated browser sessions
Remote browser support - Connect to Browserless, external Chrome, etc.
Playwright integration - Full CDP (Chrome DevTools Protocol) access
Extension relay - Browser extension-based control for certain profiles
Snapshot & interaction - Take page snapshots, click, type, navigate

Architecture

Browser control runs as an HTTP server (src/browser/server.ts:21):

const server = await startBrowserControlServerFromConfig();
// Listens on http://127.0.0.1:<controlPort>/

Components:

Control Server - REST API for browser operations (src/browser/server.ts)
Profile Manager - Manages browser profiles and state (src/browser/profiles.ts)
Playwright Bridge - CDP connection and page control (src/browser/pw-session.ts)
Extension Relay - WebSocket bridge for browser extensions (src/browser/extension-relay.ts)

Configuration

browser:
  enabled: true
  controlPort: 8910
  
  profiles:
    default:
      enabled: true
      browser: "chromium"  # chromium, chrome, edge, firefox
      headless: false
      color: "blue"
      cdpPort: 9222
      
      # Optional remote browser
      cdpUrl: "ws://localhost:3000"  # Browserless
      attachOnly: true  # Don't launch, just attach
      
      # Custom executable
      executablePath: "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
      
    work:
      enabled: true
      browser: "chrome"
      headless: false
      color: "green"
      cdpPort: 9223

Browser Profiles

Profiles are isolated browser instances with separate:

User data directory (~/.simpleclaw/browser/<profile>)
CDP port
Extension state
Cookies, storage, history

Profile Operations

# List all profiles
curl http://localhost:8910/profiles

# Get default profile status
curl http://localhost:8910/

# Get specific profile status
curl http://localhost:8910/?profile=work

# Start profile
curl -X POST http://localhost:8910/start?profile=work

# Stop profile
curl -X POST http://localhost:8910/stop?profile=work

# Reset profile (archive user data)
curl -X POST http://localhost:8910/reset-profile?profile=work

Creating Profiles

# Create new profile
curl -X POST http://localhost:8910/profiles/create \
  -H "Content-Type: application/json" \
  -d '{
    "name": "testing",
    "color": "purple",
    "driver": "simpleclaw"
  }'

# Create remote profile (Browserless)
curl -X POST http://localhost:8910/profiles/create \
  -H "Content-Type: application/json" \
  -d '{
    "name": "remote",
    "cdpUrl": "ws://localhost:3000",
    "driver": "simpleclaw"
  }'

Deleting Profiles

curl -X DELETE http://localhost:8910/profiles/testing

Browser Operations

From src/browser/client.ts, the control API provides:

Status

type BrowserStatus = {
  enabled: boolean;
  profile: string;
  running: boolean;
  cdpReady: boolean;
  pid: number | null;
  cdpPort: number;
  cdpUrl?: string;
  chosenBrowser: string | null;
  userDataDir: string | null;
  color: string;
  headless: boolean;
  attachOnly: boolean;
};

Tabs

# List tabs
curl http://localhost:8910/tabs?profile=default

# Open new tab
curl -X POST http://localhost:8910/tabs/open \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "profile": "default"}'

# Close tab
curl -X POST http://localhost:8910/tabs/close \
  -H "Content-Type: application/json" \
  -d '{"targetId": "...", "profile": "default"}'

Snapshots

Capture page accessibility tree or AI-optimized snapshot:

# ARIA snapshot (accessibility tree)
curl -X POST http://localhost:8910/agent/snapshot \
  -H "Content-Type: application/json" \
  -d '{
    "targetId": "page-123",
    "format": "aria",
    "profile": "default"
  }'

# AI snapshot (optimized for LLMs)
curl -X POST http://localhost:8910/agent/snapshot \
  -H "Content-Type: application/json" \
  -d '{
    "targetId": "page-123",
    "format": "ai",
    "labels": true,
    "profile": "default"
  }'

AI snapshot format (src/browser/pw-role-snapshot.ts):

URL: https://example.com

[1] button "Sign In"
[2] link "Pricing"
[3] textbox "Search" (empty)
[4] heading "Welcome"

Actions

Interact with page elements:

# Click element
curl -X POST http://localhost:8910/agent/act \
  -H "Content-Type: application/json" \
  -d '{
    "targetId": "page-123",
    "actions": [
      {"type": "click", "ref": "1"}  # Click [1] button
    ],
    "profile": "default"
  }'

# Type text
curl -X POST http://localhost:8910/agent/act \
  -H "Content-Type: application/json" \
  -d '{
    "targetId": "page-123",
    "actions": [
      {"type": "fill", "ref": "3", "value": "search query"},
      {"type": "press", "key": "Enter"}
    ]
  }'

# Navigate
curl -X POST http://localhost:8910/agent/act \
  -H "Content-Type: application/json" \
  -d '{
    "targetId": "page-123",
    "actions": [
      {"type": "navigate", "url": "https://news.ycombinator.com"}
    ]
  }'

Supported action types (src/browser/routes/agent.act.ts):

click - Click element by ref
fill - Fill input/textarea
press - Press keyboard key
navigate - Navigate to URL
scroll - Scroll to element or position
hover - Hover over element
screenshot - Take screenshot

Playwright Integration

Browser control uses Playwright under the hood (src/browser/pw-session.ts):

// Launch browser with CDP
const browser = await playwright.chromium.launch({
  headless: config.headless,
  executablePath: config.executablePath,
  args: [
    `--remote-debugging-port=${config.cdpPort}`,
    `--user-data-dir=${userDataDir}`,
    ...(config.noSandbox ? ['--no-sandbox'] : [])
  ]
});

// Or connect to remote browser
const browser = await playwright.chromium.connectOverCDP(cdpUrl);

CDP Endpoints

# CDP WebSocket URL
ws://localhost:9222/devtools/browser/<id>

# CDP HTTP endpoints
http://localhost:9222/json/version
http://localhost:9222/json/list
http://localhost:9222/json/new

Extension Relay

For profiles with driver: extension, SimpleClaw provides a WebSocket relay (src/browser/extension-relay.ts):

// Browser extension connects to:
ws://localhost:8910/extension/relay?profile=default&token=...

// Extension forwards CDP messages:
ws.send(JSON.stringify({
  id: 1,
  method: "Runtime.evaluate",
  params: { expression: "document.title" }
}));

Use case: Control browsers that don’t support --remote-debugging-port (e.g., some Firefox builds, Safari).

Authentication

Browser control server supports token or password auth (src/browser/control-auth.ts):

gateway:
  auth:
    token: "${GATEWAY_AUTH_TOKEN}"
    # or
    password: "${GATEWAY_PASSWORD}"

Requests must include:

curl -H "Authorization: Bearer ${GATEWAY_AUTH_TOKEN}" \
  http://localhost:8910/

# or
curl -H "X-SimpleClaw-Password: ${GATEWAY_PASSWORD}" \
  http://localhost:8910/

Remote Browsers

Browserless Cloud

browser:
  profiles:
    cloud:
      enabled: true
      cdpUrl: "wss://chrome.browserless.io?token=${BROWSERLESS_TOKEN}"
      attachOnly: true
      driver: "simpleclaw"

Self-hosted Chrome

# Run Chrome with remote debugging
docker run -p 9222:9222 \
  browserless/chrome:latest \
  --remote-debugging-port=9222

browser:
  profiles:
    docker:
      cdpUrl: "ws://localhost:9222"
      attachOnly: true

Storage & State

Browser profiles persist state to disk:

~/.simpleclaw/browser/
  default/               # Default profile user data
    Cache/
    Cookies
    Local Storage/
    IndexedDB/
  work/                  # Work profile
  .archived/             # Reset profiles moved here
    default-2024-03-03-123456/

Storage Operations

# Save storage to JSON
curl -X POST http://localhost:8910/agent/storage/save \
  -H "Content-Type: application/json" \
  -d '{
    "targetId": "page-123",
    "key": "backup-2024",
    "profile": "default"
  }'

# Load storage from JSON
curl -X POST http://localhost:8910/agent/storage/load \
  -H "Content-Type: application/json" \
  -d '{
    "targetId": "page-123",
    "key": "backup-2024"
  }'

Storage saved to ~/.simpleclaw/browser/<profile>/storage/<key>.json

Screenshots

Capture full page or element screenshots:

curl -X POST http://localhost:8910/agent/act \
  -H "Content-Type: application/json" \
  -d '{
    "targetId": "page-123",
    "actions": [
      {
        "type": "screenshot",
        "path": "/tmp/screenshot.png",
        "fullPage": true
      }
    ]
  }'

Screenshot options:

path - Save location
fullPage - Capture entire scrollable page
clip - Crop to specific region
type - png or jpeg

Troubleshooting

Browser won’t start? Check logs:

tail -f ~/.simpleclaw/logs/gateway.log | grep browser

Verify executable:

ls /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome

CDP connection refused? Check if port is in use:

lsof -i :9222

Kill existing Chrome:

pkill -9 Chrome

Extension relay not connecting? Verify WebSocket URL:

console.log(ws.url);
// Should be: ws://localhost:8910/extension/relay?profile=...

Actions timing out? Increase timeout:

browser:
  profiles:
    default:
      timeoutMs: 60000  # 60 seconds

API Reference

Key files in src/browser/:

server.ts - Main HTTP server (src/browser/server.ts:21)
client.ts - TypeScript client library (src/browser/client.ts:1)
routes/agent.ts - Agent interaction routes
routes/agent.act.ts - Page action handlers
routes/agent.snapshot.ts - Snapshot generation
pw-session.ts - Playwright session management
profiles.ts - Profile configuration
extension-relay.ts - Extension WebSocket relay

Documentation Index

​Overview

​Architecture

​Configuration

​Browser Profiles

​Profile Operations

​Creating Profiles

​Deleting Profiles

​Browser Operations

​Status

​Tabs

​Snapshots

​Actions

​Playwright Integration

​CDP Endpoints

​Extension Relay

​Authentication

​Remote Browsers

​Browserless Cloud

​Self-hosted Chrome

​Storage & State

​Storage Operations

​Screenshots

​Troubleshooting

​API Reference