Researchers from Salesforce and the University of Southern California have innovated a technique for computer-use agents that involves executing code while interacting with graphical user interfaces (GUIs). This allows agents to write scripts while navigating and clicking within applications, enhancing efficiency and minimizing errors. This hybrid system, named CoAct-1, surpasses existing benchmarks, accomplishing complex computer tasks with fewer steps than other methods. The system, comprising three agents—a Planner, a Programmer, and a GUI Operator—integrates coding for backend tasks and GUI interaction for frontend tasks. Tested on OSWorld, CoAct-1 emerged with a higher success rate while requiring fewer actions, indicating a robust path toward scalable agent automation with considerable enterprise applications. The methodology underscores a potential for automating multitool enterprise processes, addressing challenges of UI unpredictability, and maintaining security through sandboxing, possibly requiring human oversight for high-stakes operations.
