TUI Goggles

A tool that allows LLMs and automated systems to "see" TUI (Text User Interface) applications by capturing their rendered output as clean text grids.

Problem Solved

When LLMs run TUI applications, they receive raw ANSI escape sequences that are difficult to interpret. This tool spawns commands in a virtual terminal, processes all escape sequences, and returns a clean text representation of what would appear on screen.

Installation

cd tui-goggles
go build -o bin/tui-goggles ./cmd/tui-goggles

Usage

tui-goggles [flags] -- command [args...]

Exit Codes

Code	Meaning
0	Success - capture completed, all assertions passed
1	General error - invalid arguments, command failed to start
2	Timeout - operation exceeded timeout
3	Assertion failed - text specified with -assert was not found
4	Command error - target command exited with non-zero status

Flags

Flag	Default	Description
-cols	80	Terminal width in columns
-rows	24	Terminal height in rows
-delay	500ms	Initial delay before capture
-stable-timeout	5s	Timeout waiting for stable screen
-stable-time	200ms	Duration screen must be stable
-wait-for	""	Wait for this text to appear before capturing
-wait-stable	false	Wait for screen to stabilize before capturing
-keys	""	Keys to send (space-separated)
-keys-stdin	false	Read keys from stdin (one per line)
-input-delay	50ms	Delay between keystrokes
-format	text	Output format: text or json
-output	""	Write output to file instead of stdout
-timeout	30s	Overall timeout for the operation
-assert		Assert text appears on screen (repeatable, exit 3 if not found)
-check		Check if text appears (repeatable, adds to JSON output, no exit change)
-capture-each	false	Capture screen after each key (returns array in JSON mode)
-trim	false	Trim trailing blank lines from output
-quiet	false	Suppress output on success (useful with -assert)
-env		Set environment variable (format: KEY=VALUE, repeatable)

Examples

# Capture initial screen of a TUI app
tui-goggles -delay 1s -- ./my-tui-app

# Capture with custom terminal size
tui-goggles -cols 120 -rows 40 -- ./my-tui-app

# Navigate a menu and capture
tui-goggles -keys "down down enter" -delay 1s -- ./my-tui-app

# Wait for specific text before capturing
tui-goggles -wait-for "Main Menu" -- ./my-tui-app

# Assert expected text is present (for automated testing)
tui-goggles -assert "Welcome" -assert "Login" -- ./my-tui-app

# Quiet mode - only exit code matters (for CI/CD)
tui-goggles -assert "Ready" -quiet -- ./my-tui-app

# Check for text presence without failing (adds to JSON)
tui-goggles -check "Error" -check "Warning" -format json -- ./my-tui-app

# Capture each step of navigation (returns array of screens)
tui-goggles -keys "down enter" -capture-each -format json -- ./my-tui-app

# Get clean JSON output with cursor position and timing
tui-goggles -format json -trim -- ./my-tui-app

# Read keys from stdin for complex sequences
echo -e "down\ndown\nenter" | tui-goggles -keys-stdin -- ./my-tui-app

# Control keystroke timing for slow apps
tui-goggles -keys "down enter" -input-delay 200ms -- ./my-tui-app

# Save output to file
tui-goggles -output screenshot.txt -- ./my-tui-app

# Pass environment variables to the command
tui-goggles -env "TERM=dumb" -env "NO_COLOR=1" -- ./my-tui-app

# Wait for screen to stabilize before capturing
tui-goggles -wait-stable -- ./my-tui-app

# Use with piped input (e.g., fzf)
echo -e "apple\nbanana\ncherry" | tui-goggles -delay 500ms -- fzf

Key Names

For the -keys flag, use these names (space-separated):

• Navigation: up, down, left, right, home, end, pgup, pgdn
• Actions: enter, tab, esc, backspace, delete, space
• Function keys: f1 through f12
• Ctrl combinations: ctrl-a through ctrl-z
• Literal text: Any other string is sent as-is

JSON Output Format

Single capture (-format json):

{
  "screen": "...",
  "cols": 80,
  "rows": 24,
  "cursor_row": 0,
  "cursor_col": 0,
  "cursor_visible": true,
  "timestamp": "2024-01-15T10:30:00Z",
  "command": "my-app --flag",
  "checks": {"Login": true, "Error": false},
  "timing": {
    "total_ms": 1250,
    "delay_ms": 500,
    "stabilize_ms": 200,
    "wait_for_text_ms": 350,
    "keys_ms": 200
  }
}

Multi-capture (-format json -capture-each):

{
  "captures": [
    {"screen": "...", "cursor_row": 0, "cursor_col": 0, ...},
    {"screen": "...", "cursor_row": 1, "cursor_col": 0, ...}
  ],
  "command": "my-app --flag",
  "timing": {...}
}

How It Works

1. Creates a PTY (pseudo-terminal) to run the target command
2. Uses vt10x to emulate a VT100/xterm terminal and process escape sequences
3. Responds to terminal queries (DSR, DA1, OSC color queries) so applications like Bubble Tea can properly initialize
4. Captures the virtual terminal buffer as a text grid

Compatibility

Tested Applications

Application	Framework	Status
Bubble Tea apps	Go (charmbracelet/bubbletea)	Works
fzf	Go	Works
htop	ncurses	Works
top	ncurses	Works
nano	ncurses	Works
Midnight Commander	ncurses/S-Lang	Works

What Works Well

• Text-based TUI apps - menus, lists, forms, editors
• ncurses applications - the vast majority of terminal apps
• Bubble Tea / bubbletea apps - Go TUI framework
• CLI tools with formatted output

Known Limitations

Graphics protocols are not supported:

• Sixel graphics - will show blank or escape codes
• Kitty graphics protocol - inline images won't render
• iTerm2 inline images - won't render

Terminal queries we respond to:

• ESC[5n / ESC[6n (DSR - device status, cursor position) - via vt10x
• ESC[c / ESC[0c (DA1 - primary device attributes)
• ESC[>c / ESC[>0c (DA2 - secondary device attributes)
• ESC[14t / ESC[18t / ESC[19t (XTWINOPS - window/screen size)
• ESC]10;? / ESC]11;? (foreground/background color)

Queries we don't handle (may affect some apps):

• Kitty keyboard protocol
• DECRQSS (request settings)
• Some advanced xterm extensions

Other limitations:

• Complex Unicode (combining characters, wide chars) may have issues
• Right-to-left text not fully supported
• Timing-sensitive animations may not capture correctly

When to Use an Alternative

For apps requiring maximum compatibility (graphics, advanced terminal features), consider using something like mcp-tui-driver which uses the more complete wezterm-term emulator

Technical Details

Terminal Query Handling

Many modern TUI applications (especially Bubble Tea) send queries to detect terminal capabilities:

• DSR (Device Status Report): ESC[6n - Asks for cursor position
• DA1 (Device Attributes): ESC[c - Asks for terminal type
• OSC 10/11: Background/foreground color queries

This tool intercepts these queries and responds appropriately, allowing applications to complete their initialization and render properly.

Dependencies

• github.com/creack/pty - PTY handling
• github.com/hinshun/vt10x - VT100 terminal emulation

Use Cases

• LLM-driven testing: Let AI assistants verify TUI application states
• Automated testing: Capture and compare TUI screenshots in CI/CD
• Documentation: Generate text-based screenshots for docs
• Accessibility: Convert visual TUI state to text for screen readers

License

MIT