Peekaboo：面向 AI Agent 的 macOS UI 自动化工具 - Openclaw Skills

什么是 Peekaboo？

Peekaboo 是一款专为全方位 macOS UI 自动化量身定制的强大命令行界面，是 Openclaw Skills 中需要桌面交互功能的关键桥梁。它允许用户和 AI Agent 精确地捕获和检查屏幕、定位特定的 UI 元素，并管理应用程序窗口或系统菜单。通过提供快照缓存和结构化的 JSON 输出，它简化了涉及视觉识别和程序化输入模拟的复杂工作流脚本编写过程。

Peekaboo 专为现代开发工作流设计，可与基于视觉的 AI 模型无缝集成。它可以提取帧、标注 UI 映射，并执行诸如点击、拖动和输入等高级命令。无论您是在构建自动化测试套件，还是赋予 AI 导航本地环境的能力，Peekaboo 都能在 Openclaw Skills 生态系统中提供实现复杂 macOS 控制所需的底层钩子。

下载入口:https://github.com/openclaw/skills/tree/main/skills/steipete/peekaboo

安装与下载

1. ClawHub CLI

从源直接安装技能的最快方式。

npx clawhub@latest install peekaboo

2. 手动安装

将技能文件夹复制到以下位置之一

全局模式 ~/.openclaw/skills/ 工作区 /skills/

优先级：工作区 > 本地 > 内置

3. 提示词安装

将此提示词复制到 OpenClaw 即可自动安装。

请帮我使用 Clawhub 安装 peekaboo。如果尚未安装 Clawhub，请先安装（npm i -g clawhub）。

Peekaboo 应用场景

• 构建能够导航并与任何 macOS 应用程序交互的自主 AI Agent。
• 自动化重复的桌面工作流，如表单填写、数据录入或环境配置。
• 生成带有标注的屏幕截图和 UI 元数据，用于训练或调试视觉模型。
• 通过终端命令控制菜单栏、Dock 和调度中心（Spaces）等系统级功能。
• 使用类人交互模式对 macOS 软件执行自动化 UI 测试。

Peekaboo 工作原理

1. 通过验证必要的 macOS 屏幕录制和辅助功能权限来初始化环境。
2. 使用 see 命令生成当前屏幕或特定窗口的快照，并生成标注过的 UI 映射。
3. 使用快照元数据中提供的元素 ID、坐标或特定于应用程序的查询来识别目标。
4. 执行 click（点击）、drag（拖动）或 type（输入）等交互命令，在目标元素上执行所需操作。
5. 利用 --json 等全局标志，将执行结果通过管道传输到其他工具或 AI 推理循环中，实现多步自动化。

Peekaboo 配置指南

要开始使用 Openclaw Skills 库中的此工具，请通过 Homebrew 安装：

brew tap steipete/tap/peekaboo
brew install peekaboo
# 验证权限
peekaboo permissions

安装完成后，您可以使用以下命令验证连接并查看可用工具：

peekaboo bridge
peekaboo tools

Peekaboo 数据架构与分类体系

Peekaboo 通过快照缓存和结构化 CLI 输出组织数据。关键数据特性包括：

数据类型	描述
快照 (Snapshots)	以 JSON 或图像文件 (PNG/JPG) 形式存储的缓存 UI 状态。
UI 映射 (UI Maps)	带有元素 ID（如 B1, T2）的屏幕标注视觉表示。
JSON 输出	应用程序、窗口和菜单项的标准机器可读数据。
配置 (Config)	用于管理服务商、模型和凭据的本地配置文件。

name: peekaboo
description: Capture and automate macOS UI with the Peekaboo CLI.
homepage: https://peekaboo.boo
metadata: {"clawdbot":{"emoji":"??","os":["darwin"],"requires":{"bins":["peekaboo"]},"install":[{"id":"brew","kind":"brew","formula":"steipete/tap/peekaboo","bins":["peekaboo"],"label":"Install Peekaboo (brew)"}]}}

Peekaboo

Peekaboo is a full macOS UI automation CLI: capture/inspect screens, target UI elements, drive input, and manage apps/windows/menus. Commands share a snapshot cache and support --json/-j for scripting. Run peekaboo or peekaboo --help for flags; peekaboo --version prints build metadata. Tip: run via polter peekaboo to ensure fresh builds.

Features (all CLI capabilities, excluding agent/MCP)

Core

• bridge: inspect Peekaboo Bridge host connectivity
• capture: live capture or video ingest + frame extraction
• clean: prune snapshot cache and temp files
• config: init/show/edit/validate, providers, models, credentials
• image: capture screenshots (screen/window/menu bar regions)
• learn: print the full agent guide + tool catalog
• list: apps, windows, screens, menubar, permissions
• permissions: check Screen Recording/Accessibility status
• run: execute .peekaboo.json scripts
• sleep: pause execution for a duration
• tools: list available tools with filtering/display options

Interaction

• click: target by ID/query/coords with smart waits
• drag: drag & drop across elements/coords/Dock
• hotkey: modifier combos like cmd,shift,t
• move: cursor positioning with optional smoothing
• paste: set clipboard -> paste -> restore
• press: special-key sequences with repeats
• scroll: directional scrolling (targeted + smooth)
• swipe: gesture-style drags between targets
• type: text + control keys (--clear, delays)

System

• app: launch/quit/relaunch/hide/unhide/switch/list apps
• clipboard: read/write clipboard (text/images/files)
• dialog: click/input/file/dismiss/list system dialogs
• dock: launch/right-click/hide/show/list Dock items
• menu: click/list application menus + menu extras
• menubar: list/click status bar items
• open: enhanced open with app targeting + JSON payloads
• space: list/switch/move-window (Spaces)
• visualizer: exercise Peekaboo visual feedback animations
• window: close/minimize/maximize/move/resize/focus/list

Vision

• see: annotated UI maps, snapshot IDs, optional analysis

Global runtime flags

• --json/-j, --verbose/-v, --log-level
• --no-remote, --bridge-socket

Quickstart (happy path)

peekaboo permissions
peekaboo list apps --json
peekaboo see --annotate --path /tmp/peekaboo-see.png
peekaboo click --on B1
peekaboo type "Hello" --return

Common targeting parameters (most interaction commands)

• App/window: --app, --pid, --window-title, --window-id, --window-index
• Snapshot targeting: --snapshot (ID from see; defaults to latest)
• Element/coords: --on/--id (element ID), --coords x,y
• Focus control: --no-auto-focus, --space-switch, --bring-to-current-space, --focus-timeout-seconds, --focus-retry-count

Common capture parameters

• Output: --path, --format png|jpg, --retina
• Targeting: --mode screen|window|frontmost, --screen-index, --window-title, --window-id
• Analysis: --analyze "prompt", --annotate
• Capture engine: --capture-engine auto|classic|cg|modern|sckit

Common motion/typing parameters

• Timing: --duration (drag/swipe), --steps, --delay (type/scroll/press)
• Human-ish movement: --profile human|linear, --wpm (typing)
• Scroll: --direction up|down|left|right, --amount , --smooth

Examples

See -> click -> type (most reliable flow)

peekaboo see --app Safari --window-title "Login" --annotate --path /tmp/see.png
peekaboo click --on B3 --app Safari
peekaboo type "[email protected]" --app Safari
peekaboo press tab --count 1 --app Safari
peekaboo type "supersecret" --app Safari --return

Target by window id

peekaboo list windows --app "Visual Studio Code" --json
peekaboo click --window-id 12345 --coords 120,160
peekaboo type "Hello from Peekaboo" --window-id 12345

Capture screenshots + analyze

peekaboo image --mode screen --screen-index 0 --retina --path /tmp/screen.png
peekaboo image --app Safari --window-title "Dashboard" --analyze "Summarize KPIs"
peekaboo see --mode screen --screen-index 0 --analyze "Summarize the dashboard"

Live capture (motion-aware)

peekaboo capture live --mode region --region 100,100,800,600 --duration 30 \r
  --active-fps 8 --idle-fps 2 --highlight-changes --path /tmp/capture

App + window management

peekaboo app launch "Safari" --open https://example.com
peekaboo window focus --app Safari --window-title "Example"
peekaboo window set-bounds --app Safari --x 50 --y 50 --width 1200 --height 800
peekaboo app quit --app Safari

Menus, menubar, dock

peekaboo menu click --app Safari --item "New Window"
peekaboo menu click --app TextEdit --path "Format > Font > Show Fonts"
peekaboo menu click-extra --title "WiFi"
peekaboo dock launch Safari
peekaboo menubar list --json

Mouse + gesture input

peekaboo move 500,300 --smooth
peekaboo drag --from B1 --to T2
peekaboo swipe --from-coords 100,500 --to-coords 100,200 --duration 800
peekaboo scroll --direction down --amount 6 --smooth

Keyboard input

peekaboo hotkey --keys "cmd,shift,t"
peekaboo press escape
peekaboo type "Line 1
Line 2" --delay 10

Notes

• Requires Screen Recording + Accessibility permissions.
• Use peekaboo see --annotate to identify targets before clicking.