Most brokers can reply to a immediate, however ask them to click on a button in your enterprise software program, and all of the sudden its limitations present.
Within the age of generative AI, everybody’s racing to construct brokers that don’t simply reply to prompts, however really do issues. Ship an e mail. Replace a file. Navigate a dashboard. The dream, proper? An clever assistant that makes use of your apps identical to a human would, all clicks, scrolls, and savvy shortcuts.
However right here’s the catch: most AI Brokers collapse the second they contact a graphical consumer interface (GUI). Why? As a result of clicking round a display in the actual world isn’t as simple because it sounds. Enterprise software program is dense, dynamic, and sometimes irritating for people, not to mention for giant language fashions (LLMs) making an attempt to drive with verbal or pure language alone.
That’s the place Laptop Use Brokers (CUAs) are available and why Salesforce AI is utilizing reinforcement studying to enhance this know-how.
Most LLM-based brokers are constructed for language. They perceive prompts and might reply questions; nevertheless, their limitation exhibits when asking them to carry out a multi-step process inside an actual software.
Think about this state of affairs: when navigating a CRM system, a human doesn’t simply “know” what to do. They see the display, acknowledge visible cues, bear in mind previous steps, make choices in real-time, and comply with workflows that aren’t all the time apparent. An AI Agent replicating that habits requires greater than textual content prediction. It requires embodied intelligence or an understanding of it’s surroundings.
Most generic brokers fail for 2 causes:
1. Ambiguous Planning
There’s not often one “proper” solution to full a process. Ought to the agent click on the blue button or use the dropdown in your CRM? Ought to it search or scroll? Many attainable sequences would possibly work, however some are sooner, safer, or extra aligned with enterprise logic. Selecting correctly, with out hindsight, is hard. It’s the sort of decision-making people do with out pondering, however for AI it’s a high-stakes guessing sport.
2. Visible Grounding
Most UIs aren’t static or easy. Buttons transfer. Screens resize. Parts overlap. The agent has to know precisely the place to click on, and clicking the mistaken place can crash a workflow. It’s like navigating a maze the place the partitions preserve transferring.
To deal with these challenges, our Salesforce Analysis crew launched GTA1 (GUI Take a look at-time Agent 1), a cutting-edge, two-part structure designed to deal with each clever planning and exact visible grounding throughout dynamic, real-world interfaces.
At its core, GTA1 blends two important improvements:
- Take a look at-Time Scaling (Smarter Planning)
Reasonably than committing to a single motion, GTA1 samples a number of potential subsequent steps. It then evaluates them utilizing a multimodal decide mannequin (which sees and understands each the display and process context) to pick out the most effective transfer — all at runtime.
This adaptive planning system permits GTA1 to keep away from early errors and modify course on the fly, with out requiring lookahead or brittle hardcoded sequences.
- RL-Primarily based Grounding (Higher Clicking)
As an alternative of making an attempt to foretell the precise middle of a button — like many supervised fashions do — GTA1 makes use of reinforcement studying to click on anyplace inside the proper goal. The reward? Touchdown contained in the clickable zone. That’s it.
This easy however highly effective change improves flexibility and generalization, particularly in high-resolution, cluttered UIs the place “middle” isn’t all the time dependable. It additionally takes away the necessity for verbose “reasoning” earlier than clicking — one thing our analysis exhibits usually hurts grounding efficiency in static environments.
The Outcomes: Smoother Clicks, Smarter Actions
GTA1 units new requirements throughout trade benchmarks — proving that scalable, high-performing GUI brokers are now not theoretical.
📊 ScreenSpot-Professional (skilled enterprise UIs):
GTA1-7B achieves 50.1%, outperforming many fashions with 10x the parameters.
GTA1-72B scores 94.8%, rivaling prime proprietary programs.
💻 OSWorld-G (Linux environments):
GTA1-7B leads with 67.7%, excelling in textual content matching, factor recognition, format understanding, and fine-grained manipulation.
On the complete OSWorld benchmark, GTA1-7B completes 53.1% of real-world duties — beating OpenAI’s CUA o3 (42.9%) in half the steps (100 vs. 200).
And GTA1’s benefits compound when scaled. With bigger fashions and extra candidate actions (by way of test-time scaling), efficiency continues to climb — with out bloating wall-clock time because of concurrent sampling.
Laptop Use Brokers like GTA1 are constructed to do what most brokers can’t: function software program within the wild. Which means they will…
- Full precise workflows throughout CRM, ERP, or productiveness instruments, no APIs required
- Adapt to UI modifications, variations, or user-specific layouts
- Be taught from earlier interactions to enhance accuracy and pace
- Respect enterprise constraints, insurance policies, and information entry guidelines
- For Salesforce, this implies a future the place brokers can do greater than summarize information or draft emails. They will take motion, schedule a gathering, replace a pipeline, create a dashboard — all whereas grounded in our platform’s safety and belief.
Belief and Management Nonetheless Matter. Even the neatest agent wants a supervisor. At Salesforce, we’re not simply constructing brokers — we’re constructing programs with governance, transparency, and human oversight inbuilt. That’s why each CUA we construct is designed with:
- Judgment fashions for safer decision-making
- Zero-copy information entry to reduce danger and maximize context
- Observability instruments so admins and customers can monitor what brokers do — and the way effectively they’re doing it
- Belief Layer protections to implement role-based entry, compliance, and consumer intent at each click on
The subsequent technology of AI received’t dwell in chat home windows — it’ll dwell in your software program. Brokers that work throughout tabs. That understands your workflows. That really will get issues completed.
GTA1 proves it’s attainable. It’s not a demo. It’s not a dream. It’s a basis for scalable, reliable AI that clicks, scrolls, and performs — identical to an important teammate would.
