A computer-using agent (CUA) is a multimodal, model-based agent that operates graphical user interfaces as a person would: it interprets a screen, moves a cursor, clicks, enters text, observes the updated interface, and chooses the next action. Unlike API-based agents, it can work through existing applications without requiring a new information-sharing protocol or dedicated tools. section 8.15 section 8.3
CUAs are useful for workflows that require several interface actions, but their flexibility creates important limitations:
- They can click, type, navigate websites, fill forms, retrieve information, book appointments, and complete other procedural tasks across applications. section 8.3
- They must infer available actions from pixels and interface changes rather than receiving a precisely defined set of API operations. section 8.15
- They remain experimental and error-prone; visual recognition failures can derail otherwise simple tasks. section 8.15
From Splunk queries to Operator
A GUI-native agent could open an existing Splunk platform and run a natural-language query by navigating the interface like a human, avoiding a separate MCP server. OpenAI’s Operator research preview applies the same idea to broader web tasks, using CUA to click, type, and complete workflows. section 8.15 section 8.3
Figure 8.2 Example workflow of a personal assistant coordinating a scheduling task using contextual data from messages, calendar events, and user intent
Use the scheduling workflow to visualize how a personal assistant can coordinate intent, messages, and calendar information across a multistep task. section 8.3
