Computer Use and GUI Automation Tools — Tool Use, Function Calling & MCP

Every tool this course has built so far assumes a structured interface exists: a function signature, a JSON schema, an MCP server exposing typed methods. That assumption holds for the software teams build in-house and for the growing share of SaaS products that ship an API alongside their UI. It does not hold for the internal admin panel a finance team has run unmodified since 2014, the desktop application a lab instrument vendor ships with no integration surface at all, or the government portal that only exists as a web form. When the only interface to a system is the graphical one a human would use, and no one is going to write an adapter for it, computer use is what's left.

Computer use runs a specific loop, and the loop shape is what makes it expensive compared to everything else in this course. The model is shown a screenshot of the current screen, reasons about what it sees, and returns one action: a click at a coordinate, a key press, a scroll, a short pause. The action executes, a new screenshot is taken, and the loop repeats. Lesson 5 built the tool-using loop where a single model call can request several structured tool calls and get several results back in one round trip; computer use collapses that batching entirely, because the model can't know what a click actually changed until it sees the next screenshot. One action, one round trip, one image — every single time.

That per-step cost compounds badly. A single structured tool call to a `search_orders` function might cost a few hundred tokens of JSON in each direction and complete in well under a second. A single computer-use step costs a full screenshot's worth of image tokens on the way in, a model call with vision reasoning attached, and a network round trip to whatever machine is running the target application, and it moves the task forward by exactly one click's worth of progress. A task that a real API would finish in three calls can easily take twenty or thirty screenshot-driven steps through a GUI, because a GUI wasn't designed to be operated in discrete, verifiable jumps — it was designed for a human's continuous hand-eye feedback loop, which a model has to approximate one static image at a time. This lesson treats computer use as exactly what it is: the tool of last resort, reached for only when nothing else in this course's toolbox — a real API, an MCP server, browser automation against the DOM — is an option.

Analogy🏏Cricket

🏏 Think of it like cricket: Every IPL franchise's analytics desk gets the match's ball-by-ball data through an official feed — venue, over, bowler, run outcome, wagon-wheel coordinates — arriving as structured numbers the instant the ball is dead. A team analyst at the Chinnaswamy watching RCB chase a target doesn't need to look at a television at all; the feed tells them directly that Virat Kohli just took a single off Trent Boult's fifth ball of the over. Now picture a fan running a private fantasy league with no access to that feed at all. They have exactly one channel: the broadcast on their television, the same picture everyone else is watching. To log the same delivery, they have to look at the screen, read the on-screen scoreboard graphic, work out from the graphic alone that a single was taken, write it down, and then look back at the screen to confirm the total actually changed before trusting their own note. Every single delivery costs a look, a read, a guess, and a re-check — four steps for information the analytics desk got as one clean number. Nobody chooses to run a league this way; you do it because the official feed genuinely isn't available to you, and the television picture is the only channel left. That is exactly the position computer use is in: it isn't a better way to operate software, it's the fallback for the system with no API at all, where the screen is the only channel that exists, and every step costs the extra work of reading a picture instead of receiving a fact.