ALE and the Cross-Industry Agent Benchmark: When Models Leave the Exam Paper and Sit Down at a Real Desk

Once ALE moves the exam room to a real workstation, what trips up models is often not intelligence, but licenses, pop-ups, and grading scripts.

If HLE (Humanity's Last Exam) tests whether large models can solve PhD-level Olympiad math and theoretical physics problems on scratch paper, then ALE (Agents' Last Exam), created by the Berkeley RDI lab (Dawn Song's team) in collaboration with over three hundred industry veterans,

tests something else entirely: throwing the model directly into a computer loaded with real industrial software and legacy code, giving it five hours, and seeing whether it can finish a full day's work of an office worker.

In the second half of this year, this benchmark became the favorite reference point for Silicon Valley and major Chinese tech companies when purchasing Agent data. The reason is simple: current models have long maxed out single-turn dialogue and short code tests, yet when placed into actual corporate business lines, they often cannot even troubleshoot a cluster failure, or they stall the entire workflow by clicking two wrong buttons in desktop software.

Following Berkeley's newly iterated ALE v1.1 official technical report, combined with the inside story of our frontline delivery across Coding and DevOps, CUA, scientific computing, cybersecurity, and financial and legal environments, you will find that to get models working at a real desk, what holds back the entire industry are all those engineering details that textbooks never write about.

This benchmark wasn't cooked up behind closed doors by a handful of people. As stated in the official blog, its scale covers 55 specialized occupations, over 1,500 expert tasks, and more than 300 participating industry experts; from an expert's submission to entering the task bank, every single task must pass three gates—agent-assisted automated review, engineer verification, and a quality control committee—and if rejected at any step along the way, it has to be reworked.

0101 From SWE-bench to a Living Cluster

Let's start with the most fundamental area: Coding and DevOps. The test here isn't about writing a function, but figuring out from a pile of microservices what brought production down.

Over the past two years, the most familiar approach to evaluating coding has been SWE-bench: give the model a GitHub repository and a bug description, have it modify a few dozen lines of code, and passing the unit tests counts as a win.

Yet anyone who has worked in backend engineering or SRE at a major tech company knows that the hardest part of daily work is rarely writing a specific function—it is when a production system suddenly crashes, and you have to dig through a tangle of microservices, databases, and configuration files to figure out what caused the outage.

That is exactly what the Computing & Engineering category in ALE tests. For instance, a task called k8s_payment_api_root_cause_analysis directly simulates a live incident where a production payment API has gone down.

Once inside the environment, the model faces a failing Kubernetes cluster; it has to run commands on its own to inspect Pod statuses, comb through logs, verify routing and configuration changes, and pinpoint and fix the root cause of the payment failure.

There are also cluster migration tasks like k8s_migration_1. When grading, the evaluation script does not just check whether the service is up and running—it directly compares the live Pod Spec against a reserved snapshot, verifying everything down to the Image Digest and suspension states without the slightest discrepancy.

Beyond that, tasks like model-parallel checkpoint weight consolidation in distributed training (mp_checkpoint_consolidation), feature parity recovery for recommendation system ranking nodes (ranking_node_feature_parity_recovery), and enterprise-grade data cleaning pipelines (ETL) have all been packed into the benchmark.

The hardest part of building this kind of environment is that it cannot be a static codebase—it must be a live set of services.

When we deliver such L2-level improved environments to model labs under the Harbor Task standard (corresponding to the ale-cli-bench specification), every single task must come with an independently bootable Docker environment, plus an executable reference solution (Oracle) handwritten by senior engineers.

This single hard rule is enough to filter out more than half of the small shops on the market that get by cobbling together open-source task sets—because many teams cannot even get containers with multiple dependent services to boot reliably within a fixed window every time, let alone steadily deliver 200 multi-model cross-validated original ops tasks per week.

0202 CUA: Workstations Without a Command Line

Once you step into the graphical interface, the trouble immediately doubles: a massive amount of industrial software has no command line at all.

If a model only types commands in a black-and-white Linux terminal, things are still relatively straightforward. Once it steps into CUA (directly operating a computer's graphical user interface), the engineering headaches immediately double.

In real-world industrial design and enterprise office work, a vast number of operations simply have no command-line interface. In ALE's task bank, you can see all kinds of desktop software tasks: handling end-to-end enterprise resource planning workflows in Odoo, building business intelligence dashboards in Metabase,

performing brand visual revamps in PowerPoint (where the grading script even compares regions of the screen to check whether the layout is crooked), drawing PCB layouts for navigation switches in KiCad, reconstructing 3D architectural models from 2D engineering drawings in Blender, generating G-code toolpaths for CNC machining in FreeCAD and LinuxCNC,

and even transcribing sheet music and migrating projects in professional audio software like MuseScore and Ardour.

To allow models to both type commands and click the mouse, the ALE team built a dual-bridge architecture called ALE-Claw: when the model runs outside the sandbox, it connects into the virtual machine through two MCP bridges—one managing the command line and file read/write operations, and the other dedicated to GUI screenshots, mouse clicks, and keyboard inputs.

It sounds smooth, but when the ALE team upgraded to version v1.1 in the second half of this year, they hit a major pitfall due to software environment issues.

In the initial version, some experts conveniently used proprietary commercial software to create tasks. As a result, when it came to large-scale evaluation and reinforcement learning training, licenses became a dead end: running a single evaluation or a round of RL at a model lab easily requires spinning up hundreds or thousands of sandbox instances simultaneously. Commercial software licensing mechanisms simply do not allow this, nor can the environments be freely distributed.

Ultimately, left with no other choice, the Berkeley team bit the bullet in v1.1 and completely rewrote 6 tasks that relied on commercial software, migrating them to open-source Linux software;

There was also an engineering task on 4-cavity injection molding simulation (mold-flow), which had to be painfully removed from the active task set altogether because the open-source community simply lacked an alternative software capable of accurately simulating the entire process of filling, packing, cooling, and ejection simultaneously.

Even after switching entirely to open-source software, standard containers often fail to hold up.

For instance, the highway alignment design task in ALE (cailian_road_highway_alignment_2) relies heavily on systemd and dedicated control groups (cgroups) for underlying graphics and process management. Running it in a standard Docker container causes it to crash immediately; spinning up a full Linux virtual machine is mandatory.

Putting these three factors together, what v1.1 actually changed is the environment distribution method, not the difficulty of the tasks. Commercial software licensing bottlenecks concurrency, systemd and cgroups bottleneck container boundaries, and auto-update pop-ups bottleneck UI operations—none of these three are listed on a model's capability sheet, yet each can reduce an evaluation run straight to zero.

This is also why we built a standardized VM image pipeline specifically when delivering ALE and CUA environments for major tech companies. Whether it is Windows 11 Enterprise or Ubuntu 24.04 Desktop, everything is packaged into a 60G monolithic single-file VMDK image with at least 20G of free headroom, booting via BIOS by default.

In Windows, VirtIO drivers are pre-installed and WinRM and SSH are pre-configured; on Ubuntu Desktop, we forcibly switch back to the stable X11 protocol (avoiding the black-screen screenshot pitfalls of Wayland), and disable Swap along with all automatic update pop-ups.

Often, when a model scores zero in a CUA evaluation, it is really not because the model is dumb, but because a system update prompt popped up and blocked the button.

0303 Scientific Computing and Real-World Offense & Defense

Venturing deeper into the weeds, the scientific computing and security tasks are pulled straight from real laboratories and live network traffic.

Once the underlying virtual machines and desktop environments are sorted out, venturing deeper leads to Scientific Computing (Science) and Cybersecurity (Cyber).

Previously, many assumed AI for Science merely meant answering biomedical questions or reciting chemical formulas from memory. In ALE's Science track, however, the tasks are computational workflows pulled straight out of real research laboratories.

In physics and astronomy, for example, a task named hst_acs_wfc_visit_reduction requires the model to directly process raw exposure data from the Hubble Space Telescope's Advanced Camera for Surveys (ACS/WFC), performing end-to-end denoising, calibration, and star map mosaicking;

In condensed matter physics, models must orchestrate computational materials science toolchains to calculate the excitonic absorption spectrum of molybdenum disulfide (MoSe2) under spin-orbit coupling (mose2_bse_absorption_soc), or complete band structure and dielectric function convergence runs for silicon; in life sciences and medicine,

models not only have to run whole-genome variant detection (WGS Variant Calling), execute three-stage molecular dynamics binding free energy calculations (MM-GBSA) with AMBER, and score HIV dimer interfaces with ZDOCK, but even review chest X-ray annotations in MicroDICOM,

or reproduce intensity-modulated radiation therapy (IMRT) dose planning for prostate cancer using matRad.

Meanwhile, in cybersecurity (Cyber), the tide has shifted completely toward real-world offense and defense.

Just in early November, OpenAI disclosed a set of figures that shook the security community: starting from 2,308 candidate vulnerabilities, an internal security model dynamically reproduced 1,724 historical vulnerability environments and followed the trail to uncover 948 previously unknown local privilege escalation (LPE) vulnerabilities in the Linux kernel, spending roughly $1.2 million in API costs across the entire process.

Among these three numbers, the most noteworthy isn't 948, but the drop from 2,308 to 1,724. Turning a vulnerability advisory into a reproducible environment that can be repeatedly attacked and automatically graded is the most expensive step in itself. The difficulty of security tasks never lies in knowing the vulnerability ID, but in whether it can be reconstructed into a working machine.

If you look at the security tasks in ALE, they follow the exact same playbook.

In ghidra_malware_config_extraction_01, the model has to open Ghidra, the open-source reverse engineering tool from the NSA, decompile a real malware binary, and follow assembly instructions and cross-references to extract the hidden command-and-control (C2) server configuration;

In pcap_enterprise_triage_01, the model is faced with a massive packet capture (PCAP) of enterprise internal network traffic, and must act like a security incident response engineer to sift out the hacker's intrusion chain from a sea of requests;

And in tasks like tris_crackme and newyear_keygen2, the model is directly asked to reverse-engineer and crack binary programs and write keygens.

This is also where our team invested the most effort when building out our two security product lines, exploitbench and cyberbench. When creating this kind of data, you can neither pad it out with old CTF problems from the internet that models have long since memorized, nor fail to strictly seal the vulnerability-triggering environment inside offline containers. One misstep, and you either fail to train real-world capability or end up with a security incident.

0404 Finance and Law: Accounting Definitions Cost More Than Prose

In the area closest to commercial monetization, what's valuable isn't writing smoothly, but getting the underlying definition of every single number right.

The final tough nut to crack—and the one closest to commercial monetization—is professional white-collar work like finance and law.

Many people think finance and law are just about reading PDFs and writing summaries, which large models excel at. However, if you actually take a look inside an investment bank, a law firm, or the Big Four accounting firms, you will understand that what makes professionals valuable is never just writing smoothly—it is reconciling the exact definition of every number and digging out every hidden clause conflict across dozens of contradicting documents and tens of thousands of spreadsheet rows.

In ALE's business, finance, and legal directories, the tasks are all high-stakes jobs like these: parsing SEC 10-K annual reports, building equity research valuation models across multiple worksheets, pricing American options using the Longstaff-Schwartz Monte Carlo method, calculating operational risk capital for commercial banks under the Basel Accords, writing algorithmic trading strategies for A-share northbound capital,

or conducting cross-document consistency audits across more than a dozen legal documents in mergers and acquisitions (M&A) transactions (legal_ma_consistency_audit_01).

In the legal domain, another widely recognized industry benchmark is Harvey LAB (Legal Agent Benchmark), created by Harvey, the legal AI unicorn valued at ten billion dollars.

It covers 1,250 real law firm tasks across 24 legal practice areas worldwide, backed by more than 75,000 grading criteria (Rubrics) handwritten item by item by senior attorneys.

Why do evaluations in professional domains have to make the grading criteria so granular? Because large models are far too good at using polished jargon to cover up miscalculated numbers.

When our team was delivering this type of long-horizon financial and legal test set (including a professional evaluation pipeline benchmarked against Harvey LAB specifications with a weekly capacity of 200 tasks), we encountered a classic case.

At the time, we created a due diligence task using 33 real bankruptcy reorganization announcements from a listed company, centering on a single core question: What is the actual recovery rate for ordinary creditors?

In the chain of hard evidence extracted by human experts, the reorganization plan stipulated that the portion of each creditor's claim under 100,000 yuan would be repaid in full, while the portion exceeding 100,000 yuan would be repaid according to different schemes. Adding the full payouts for small claims to the excess portion and dividing by the total confirmed amount of ordinary claims yields a true recovery rate of 35.36%.

However, when we tested several leading large models on the market, nearly all of them failed completely. Some simply copied an estimated figure of 41.65% from a corner of an announcement to turn in, while others were misled by the 100% recovery rate for the small-claim portion.

Even worse, when using a standard LLM Judge to grade the responses, the judge model saw that they cited the original announcements and used highly professional formatting, and ended up giving them all high scores.

For the very same problem, different scoring rubrics can lead to completely opposite conclusions. Human experts calculate the weighted actual recovery rate, while the model simply copies the most prominent estimated figure from the announcement; an LLM Judge checks whether citations are neat and formatting is professional, yet fails to notice the underlying data definitions. Grading itself has become a task that requires evaluation.

To fix this flaw, when writing Rubrics for such professional tasks, we set a strict rule: besides positive scoring criteria, we must include Negative Rubrics specifically designed to catch hallucinations and definitional errors.

Whenever the report submitted by a model contains misleading, unweighted figures like 41.65%, or—as we tested in our medical epidemiological investigation workflow (DM-MED-WORKFLOW)—fabricates monitoring indicators never mentioned in the five original source documents and writes them into the Word and Excel deliverables, it immediately triggers point deductions, no matter how well-written the overall report is.

0505 Task Creators Must Cut Their Own Questions First

Tying these five directions together, what ALE is truly doing is pouring a bucket of cold water on an Agent industry that has been racing ahead for two years.

Not long ago, the Berkeley RDI team conducted an extremely rigorous internal overhaul of the initial ALE task pool: they scrutinized all 152 existing tasks one by one, ultimately eliminating 30 tasks in one go. Among the remaining 122 active tasks, 57 were forced to undergo major revisions—including 47 where ambiguous instructions were modified,

43 where flawed verification scripts (Verification) were rewritten, and 32 where environment configurations were fixed.

Cutting a fifth and heavily revising half of a flagship benchmark they had just released—why was Berkeley so ruthless with its own work?

Among these 57 revised tasks, modifying instructions, rewriting verification scripts, and fixing environments each accounted for a share, indicating that the issue rarely lies in tasks being too hard or too easy, but rather in whether models can be scored unambiguously. If a task has ambiguous instructions, a strong model that solves it correctly based on its own interpretation might be marked wrong; if the verification script leaves loopholes, a weak model can also get full marks.

The credibility of a benchmark depends on the rigor at both ends, not on the number of tasks.

Because they are the exact same people who previously exposed widespread cheating vulnerabilities in SWE-bench, Terminal-Bench, and OSWorld. They know better than anyone that if even a tiny backdoor is left in the environment, models trained with reinforcement learning will squeeze right through the cracks.

So if you look at ALE's current execution mechanism (ale_run), it is designed in a very interesting way: the system first spins up the sandbox, places the task input inside, and then lets the Agent go in to do its work; only after the Agent has completely finished running and handed over control does the system mount the Hidden Reference into the container and launch the grading script to score it.

Turn in the test paper first, clear the room, and only at the very end bring the answer key into the exam hall for grading.

This seemingly simple timing change directly shuts down the most common cheating tactics used by large models over the past two years—whether it is digging through local Git history, searching for hardcoded numbers in test scripts, or modifying curl and dynamic link libraries in the system, as long as the reference answer does not exist in the virtual machine at all during runtime, the model has no way to peek.

There is another set of comparisons in the paper worth looking at together: fixing the model and swapping the toolchain results in a pass rate difference of 4.9 to 7.2 percentage points; fixing the toolchain and swapping the model results in a difference of 16.8 percentage points. The toolchain can indeed eat up a chunk of the score, but the gap between the models themselves is even larger. Making the evaluation environment solid is about measuring those 16.8 percentage points accurately, not about padding the scores for the models.

Ultimately, whether it is Berkeley's sweeping overhaul of ALE v1.1 today or major model labs tightening their acceptance standards, none of it is meant to deliberately make things difficult for anyone—it is a natural reaction forced by the inflation in evaluation systems over the past year. When simple single-turn questions are maxed out and static rubrics have their patterns figured out by models, the entire industry has no choice but to move the exam room into real, heavy, and strictly isolated production-grade sandboxes.

Yet if you think deeper about this, it still cannot escape the cold reality of Goodhart's Law: when a measure becomes a target, it ceases to be a good measure.

Even though ALE has now locked reference answers outside the container and turned 122 industrial-grade tasks into an anti-cheating fortress, once this public question bank becomes the mandatory report card every model lab shows off when releasing a new model, targeted cramming and isomorphic data synthesis around these 55 specialized occupational scenarios will immediately roll out.

And for teams that truly want to train capable models, those 122 tasks on the public leaderboard will always be just an entry ticket—whoever can continuously turn real-world cluster failures, software interactions, scientific computations, and contract pitfalls that have not yet been written into public exams into clean, self-contained private environments with executable verifiers is the one truly holding the ticket to the next generation of models.

📚 Image Sources

[1] Agents' Last Exam, Berkeley RDI, arXiv:2606.05405, Figure 4 (2026-06-03)

[2] Agents' Last Exam, Berkeley RDI, arXiv:2606.05405, Figure 6 (2026-06-03)

[3] Agents' Last Exam, Berkeley RDI, arXiv:2606.05405, Figure 12 (2026-06-03)

About DataMining Founded by a team of PhD students from the Institute of Software, School of Computer Science at Peking University, we focus on providing high-quality original task data and executable evaluation environments for LLM and agent R&D. We deliver not only data, but also industrial-grade Rubric and Harbor Task verification systems. ▸ Long-Horizon Code & Engineering Simulation: Deep SWE, Frontier SWE, Frontier-Eng, ALE / Terminal-Bench, etc., supporting stable delivery of 200 tasks/week within 7 business days. ▸ Extreme Math/Science & Professional Agents: HLE strict multimodal pipeline (5000+ off-the-shelf tasks); Harvey LAB, Automation Bench, Toolathlon, and 35+ atomic-level Rubrics industry workflow packages. ▸ System Infrastructure & Embodied Real-Robot Data: Standardized Win11 / Ubuntu desktop Agent VM images, as well as million-scale customized real-robot pipelines for embodied AI Grounding / SFT / DAgger and cyber data. All benchmarks, figures, and official releases cited in this article are public third-party works; capability assessments discussed herein are our internal conclusions under unified criteria and do not represent official third-party conclusions. 🌐 Website: datamining.cloud | 📮 Sample Requests & Technical Inquiries: contact@datamining.cloud