Computer Use
Computer use lets a model view a screen and return actions such as clicking, typing, or scrolling. An application harness checks and performs each allowed action.
screen → action → harness → next screencurrent screen: expense.testAn agent opens a test expense app and fills in a $248 draft. It pauses before submitting and records each screen and action.screenshottype amountallow · executescreen → model action → harness check/execute → next screenOPSAnother example
An agent opens a test expense app and fills in a $248 draft. It pauses before submitting and records each screen and action.
Main parts
- 01Current screen
- 02Returned action
- 03Target and arguments
- 04Harness check
- 05Action result
- 06Approval, log, and stop
Use it when
Use it when a visual interface must be operated and no stable API fits. Add isolation, action limits, confirmation, and stop rules.
Do not use it when
Prefer a tested API when one exists. Require a person to confirm payment, deletion, secret entry, or account changes.
Name used in code
screen → model action → harness check/execute → next screen Before you ship
Check the model and prompt versions, source data, versioned evaluation set, and measures. Verify links to evidence, tool permissions, privacy, common failures, decline-to-answer and fallback behavior, human review, monitoring, cost, speed, and rollback.
Outcome: Use or evaluate Computer Use to make AI behavior measurable and tied to evidence. User context: An agent opens a test expense app and fills in a $248 draft. It pauses before submitting and records each screen and action. AI method or concept: Computer Use. Why it fits: Use it when a visual interface must be operated and no stable API fits. Add isolation, action limits, confirmation, and stop rules. Do not use it when: Prefer a tested API when one exists. Require a person to confirm payment, deletion, secret entry, or account changes. AI requirements: Define the input and source evidence. Set model and tool permissions. Use versioned evaluation data and measures. Define failure, decline-to-answer, privacy, speed, and cost limits. Operational safeguards: Keep a clear trace and hide sensitive log data. Show users a safe fallback. Mark steps that need human review. Define how to roll back the model or prompt. Acceptance criteria: Record baseline and target measures on a versioned evaluation set. Test edge cases and hostile inputs. Verify fallback, monitoring, permissions, and rollback. Evidence and limits (evidence boundary): It is official for that source. Other platforms or teams may use the term in another way. Unknowns to confirm: Target task, model and version, evaluation owner, source data, risk limit, tool permissions, and production fallback.
Check this request
Review the current use of Computer Use. Definition: Computer use lets a model view a screen and return actions such as clicking, typing, or scrolling. An application harness checks and performs each allowed action. Release checks: Check the model and prompt versions, source data, versioned evaluation set, and measures. Verify links to evidence, tool permissions, privacy, common failures, decline-to-answer and fallback behavior, human review, monitoring, cost, speed, and rollback. Before changing code, report the evidence you found, gaps, severity, and the smallest safe fix.