How Do You Know If an AI Agent Can Really Do the Work?

Make a 3 minute landscape (16:9) training video for people at work who use or buy AI tools. Explain the APEX-Agents research benchmark: what a benchmark is, worlds (complete simulated projects with many files, built by professionals in investment banking, consulting and law), tasks (480 expert written assignments of one to two hours), rubrics (true or false criteria that must all be met) and Pass at 1 (the share of tasks an agent gets fully right on the first try, about 24 percent for the best agent). Include two short pause and think checks and a closing reflection. Use clean, professional workplace scenes with simple diagrams for worlds, files, rubric checklists and a score gauge, and no real company logos.

Your team has been testing an AI assistant. It seems to work most of the time. So the plan is to give it more data and hope it gets better. Before we go on, pause the video and ask yourself: how would you know whether the AI assistant can finish a real piece of work, from start to finish, without help? That is the question a research benchmark called APEX-Agents was built to answer. APEX-Agents is a benchmark. It does not train an AI, and it is not a product. It is a test of whether an AI agent can complete realistic professional work using the same materials a person would use. The test is built from worlds. A world is a complete project, the way it would look in a real firm. Professionals in investment banking, consulting and law played out each project over several days. The result is 33 worlds, each with about 166 files: spreadsheets, documents, presentations, emails and notes. The agent cannot search the web. Everything it needs is inside the world. Quick check. Pause the video if you like. Why build a whole world instead of asking one question? Three. Two. One. Real work means finding and combining information across many files. Inside the worlds sit 480 tasks. Each one was written by an expert and would take a professional one to two hours. To finish a task, the agent has to understand the goal, find the right files, use the right tools and produce a real piece of work. Every task comes with a rubric. A rubric is a short list of criteria, each one simply true or false. And here is the hard part: the agent must meet every single criterion. Meeting three out of four is not partly right. It is a fail, because in professional work, one missing requirement makes the work incomplete. Quick check. Pause the video if you like. An agent meets three of four criteria. Did it pass? Three. Two. One. No. It failed the task. The main score is called Pass at 1. Pick a task, run the agent once, and ask: did it meet every criterion? When the benchmark was published, the best agent passed only about 24 percent of tasks on the first try. So: experts build a world, write tasks inside it and set a rubric for each one. The agent attempts the task once, and Pass at 1 tells you how often it gets the whole job right. Pause the video and finish this sentence for yourself. I used to think that if an AI worked most of the time, more data would make it reliable. Now I think… Next time someone says an AI tool "mostly works", ask how it was tested, and whether "mostly" means the whole job, or just parts of it.