Microsoft releases Fara-7B: a small on-device model that can see and operate computers, heralding a new era for AI agents.

Although the mainstream trend in AI models is currently bigger-is-better, these large models can only run in massive data centers with huge numbers of GPUs, and many sensitive data cannot be casually uploaded to cloud servers. As a result, small open-source language models that can run locally are also a goal many AI practitioners are striving toward. In 2024, Microsoft officially launched its deployment strategy for Small Language Models (SLMs), first releasing the Phi series models on Microsoft Foundry and bringing the on-device Phi Silica to Windows 11-powered Copilot+ PCs. Recently, Microsoft took another significant step forward, officially announcingComputer Use Agent (Computer Use Agent, CUA)the first SLM designed by:Fara-7BFara-7B only has 7 billion parameters, but with strong performance, able to achieve the industry’s highest performance at the same scale, and light enough to run directly on-premises, no longer needing to send all data back to the cloud.

Tech Deep Dive: Fara-7B’s Core Architecture and Operating Logic

Advantages of lightweight design and edge computing

Fara-7B has 7 billion parameters (7B), making it quite compact in today’s landscape where models often have hundreds of billions of parameters. This size advantage allows it to run directly on users’ devices, such as Copilot+ PCs equipped with NPUs. This brings two significant advantages:

• Low Latency (Reduced Latency): Eliminating the round-trip time for data transmission to the cloud makes operations more responsive.

• Privacy Protection (Improved Privacy): All inference and data processing are completed locally, so users’ sensitive data never needs to leave the device, which is crucial for handling personal account or transaction information.

 Visual Perception: “Seeing” Screens Like Humans Do

Unlike traditional automation scripts that rely on backend Accessibility Trees or HTML parsing to understand webpage structure, Fara-7B takes a purely visual perception approach. It is based on Qwen2.5-VL-7B The model is built with powerful visual grounding capabilities and a context window of up to 128k tokens.

The operational logic of Fara-7B is “Observe-Think-Act”:

• Observation: It directly captures screenshots of web pages.

• Prediction: No additional parsing model is required; it can directly predict the coordinates of the operation target.

• Action Interact with the computer by simulating mouse clicks (Click), scrolling (Scroll), and keyboard input (Type).

In the interface named Magentic-UI, the experimental case demonstrations conducted by Fara-7B fully showcase the model’s practical application scenarios: the ability to automate daily web workflows, such as travel booking, information gathering, and price comparison. The highlight is that its ultra-compact size allows it to run locally.

Buy X-Box SpongeBob-themed controller

Fara-7B can search for and purchase the Xbox SpongeBob controller in Magentic-UI, automatically pausing at key steps like checkout to ask for user permission.

Check the latest three updates from Microsoft/Magentic-UI’s GitHub and summarize them.

Demonstrate Fara-7B using Magentic-UI to search for relevant information online and summarize it, and ask Fara-7B to find and summarize the three latest issues on Microsoft/Magentic-UI on Github.

Plan the journey.

When planning an itinerary, it can combine Bing Maps to check driving times and Bing Search to find nearby cheese specialty shops, demonstrating cross-application information integration capabilities.

Knowledge Distillation from Multi-Agent Systems: How Fara-7B Was Trained

The biggest bottleneck in developing computer-using agents (CUA) lies in data. Collecting high-quality, multi-step data of human computer operations is extremely expensive and time-consuming. To address this problem, Microsoft has developed a system based on Magentic-One The framework’s synthetic data generation pipeline successfully constructed a training dataset containing 145,000 trajectories, covering 1 million steps.

The pipeline is divided into three key stages:

Task Proposal: To ensure task diversity, the system extracts URLs from real web indexes (such as shopping, travel, and restaurant sites) as “seeds.” For example, from a movie website URL, it generates a specific task like “Book two tickets for the Downton Abbey finale at AMC Union Square in New York.” The system even lets an LLM agent explore the website first, then refines the task instructions based on the information obtained.

Task Solving: This is the core generation process. Using the Magentic-One multi-agent system, an Orchestrator agent creates a plan and directs the WebSurfer agent to execute browser operations. If user input is needed, the system can also invoke the UserSimulator for multi-round interaction. These operational steps, observations, and the agents’ reasoning processes are fully recorded to form training data.

Trajectory Verification: To ensure data quality, Microsoft introduced a triple verification mechanism:

• Alignment Verifier: Check whether the operation trajectory aligns with the original task intent.
• Standard Verifier (Rubric Verifier): Score the trajectory according to the predefined completion criteria.
• Multimodal Verifier: Review the screenshots and responses to confirm that the visual evidence supports the successful completion of the task.

Benchmark Results: Redefining the Performance/Cost Curve with Lightweight Models

Microsoft evaluated Fara-7B on multiple public benchmarks, including WebVoyager, Online-Mind2Web, Deepshop, and Microsoft’s newly released WebTailBench(For benchmarks focusing on long-tail tasks such as job searching and price comparison), Fara-7B has delivered impressive performance.

Data shows that Fara-7B demonstrates state-of-the-art (SOTA) level performance across multiple metrics, even surpassing larger models or GPT-4o agent systems that rely on auxiliary markings (Set-of-Marks, SoM).

• WebVoyager Test: Fara-7B achieved 73.5% The task success rate outperforms OpenAI computer-use-preview (70.9%) and UI-TARS-1.5-7B (66.4%).

• WebTailBench Test: In this new benchmark for complex real-world tasks, Fara-7B achieves 38.4% The success rate is far higher than the 30.0% of the GPT-4o SoM Agent.

※ All results are the average of multiple experiments. 

Efficiency is another major highlight of Fara-7B. In comparisons among models at the 7B parameter level, although Fara-7B and UI-TARS-1.5-7B share the same inference pricing (based on the Qwen2.5-VL-7B architecture), Fara-7B requires on average only 16 steps It can complete the task in just a few steps, while competitors require about 41 steps. This means that in practical applications, Fara-7B can achieve the goal with less time and computational cost, breaking new ground on the cost-effectiveness curve (Pareto Frontier).

Safe and Responsible AI: A Safety Net for Human-Machine Collaboration

Giving AI the power to operate computers comes with risks, such as operational errors, privacy leaks, or malicious exploitation. In designing Fara-7B, Microsoft emphasized the importance of “transparency” and “user control.”

Critical Decision Points (Critical Points) Mechanism

To prevent AI from performing irreversible high-risk actions (such as sending emails or confirming payments), Microsoft forcibly incorporated the concept of “key decision points” into its training data. When Fara-7B identifies such scenarios, it must stop and request user approval. This ensures that humans always retain final decision-making authority.

Sandbox Environment and Rejection Mechanism

Microsoft strongly recommends running Fara-7B in a sandboxed environment so that users can monitor and interrupt its behavior at any time. Additionally, the model has undergone rigorous red teaming and safety data training. In the WebTailBench-Refusals test for rejecting harmful tasks, Fara-7B demonstrated 82% with a high rejection rate, can effectively identify and reject malicious instructions such as jailbreak attempts, harmful content generation, or prompt injections.

Currently, Fara-7B is positioned asExperimental Preview Version (Experimental Release)，Microsoft has released it under the MIT license at Microsoft Foundry and Hugging Face Open-weight. Developers can directly download the pre-optimized models, or test them via the Magentic-UI prototype.

Microsoft’s move is intended to lower the barrier to CUA technology and invite the community to participate. Fara-7B demonstrates that through pure Supervised Fine-tuning, small models can also achieve outstanding results on complex agent tasks. Looking ahead, with the evolution of multimodal foundation models and the introduction of Reinforcement Learning techniques, AI agents that can run on edge devices will become even more powerful and reliable. The arrival of Fara-7B is not just an improvement in technical specifications, but a preview of future digital life: an era in which your computer can truly “understand” and “get done” tedious chores for you is already within reach.

Source

Source: KOCPC Chinese