前沿实验室在买什么数据？Handshake AI 深度解析
论证要点
- Handshake AI 训练业务一年内从零做到 10 亿美元年化毛收入，说明前沿实验室对专家数据的需求规模巨大。
- 金融任务中，模型能解释税务规则，却在识别具体代码行是否适用该规则时出错，暴露了规则应用与知识陈述的差距。
- 生物图像基准 VIALS 显示，模型能选出部分正确切片但会多选，专家指出组织缺失的讲解比分数更有教学价值。
- 编码安全评估中，团队检查代理读取了哪些文件、是否核对副本、如何修改 shell 历史，仅看最终回复会遗漏大部分行为。
- 评分标准（rubrics）在需要专家判断的任务上存在不足，实验室因此保留部分评估任务不公开，并为企业设计私有基准。
- 代理可能钻 Excel 评分器的空子，说明评估环境需要惩罚陷阱、依赖门和部分 credit 等机制来防止作弊。
对实践的启示
做 AI 应用时，不能只看模型能否说出规则，要测试它能否在具体情境中正确应用；评估代理需追踪工具调用全过程而非仅看输出；企业可考虑构建私有基准来捕捉专家判断的细微差别。
原文摘引
The model can explain the tax treatment when you ask it directly. The mistake comes when it has to recognize that this particular line of code is where the rule applies.
Watching Nick point to what’s missing from the tissue is more instructive than seeing the score.
A transcript of the agent’s final reply would miss most of what Steve Kelling and Nishant Ranka are evaluating.
▸ 展开原文全文（50 段，抓取自 Latent Space）
This is our first Latent Space deep dive with an AI data company. In roughly a year, Handshake’s AI training business went from launch to $1 billion in gross annualized revenue. What are frontier labs buying at that scale? What evals are models still nowhere close to human level on? Garrett Lord brought the people who build these reinforcement learning environments to show us. For the first time, we are walking through them live!
To cover the scope of Handshake, we had to produce a very special joint podcast across our Science/Engineering/Research/Finance podcast hosts, because Handshake’s business runs across all these and more.
Watch for three segments after the general intro:
- Finance. In the finance demo, the agent has to find its assignment in a bank’s email, work out which files govern the deal, and deliver a workbook. Watch what happens to one line in the client’s email about existing goodwill. The agent carries the number into Python and subtracts it. Collin points out that the model can explain the tax treatment when you ask it directly. The mistake comes when it has to recognize that this particular line of code is where the rule applies.
- Bio. Nick Magazine and Jonas Mueller show us VIALS, Handshake’s benchmark for interpreting the images scientists use at work. Nick puts nine slides of brain-tumor tissue on screen and asks which two show the most advanced vessel necrosis. Give yourself a moment before his answer. Nick picks H and I. The model picks those two and adds G. Watching Nick point to what’s missing from the tissue is more instructive than seeing the score.
- Coding. In a simulated file-access task, the team can inspect which files an agent read, whether it checked the copies, and what it did to the shell history afterward. A transcript of the agent’s final reply would miss most of what Steve Kelling and Nishant Ranka are evaluating.
Who’s in the room
- 🟣 AI data business — Garrett Lord, Sahil Bhaiwala and Nishant Ranka open and close the episode on the market for expert data.
- 🌸 Life sciences — Nick Magazine and Jonas Mueller put the gel and brain-tissue tasks on screen.
- 🟢 Finance — Collin Schweiker and Sahil Bhaiwala take us through Project Falcon and its Excel grader.
- 🟠 Alignment — Steve Kelling and Nishant Ranka follow the agent’s tool calls through the safety simulations.
Show notes
- ATLAS-Finance — Explore the benchmark for agents working across a simulated bank’s inboxes, documents, and financial models.
- Inside a bank — Handshake traces a separate published deal, Project Osprey, from the client’s email to a grading decision. The deal on screen in our interview is Project Falcon.
- VIALS sample tasks — Try the scientific image questions, including the gel and brain-tissue tasks from the episode.
- How Handshake trained a model to read scientific figures — The team describes using intermediate image descriptions to train and evaluate a vision-language model.
Timestamps
0:00 Inside Handshake AI
0:39 Garrett, Sahil and Nishant on frontier training data
5:15 How to build a banking environment for an agent
13:07 How frontier labs buy data
15:51 How a lab evaluates a new dataset
20:01 Testing training data on open models
27:38 Where rubrics fall short of expert judgment
31:30 Nick and Jonas introduce the VIALS benchmark
35:52 Reading molecular structures and plasmid maps
43:48 Training vision models to interpret scientific images
47:17 Task one: reading the bands on a gel
50:59 Task two: finding necrosis in brain tissue
55:49 Can Python tools help the model see?
1:02:52 Can biology models improve on their own?
1:06:19 What a biology lab can buy from Handshake
1:12:33 Atlas Finance with Collin and Sahil
1:18:56 Inside a simulated investment bank
1:25:45 Where the agent’s financial model went wrong
1:31:18 How a banker grades the spreadsheet
1:41:12 Penalty traps, dependency gates and partial credit
1:46:41 How an agent gamed the Excel grader
1:48:52 What would make an AI banker useful?
1:54:48 Safety and alignment with Steve and Nishant
1:58:25 CAREBench and the limits of a refusal
2:04:42 How safety feedback reaches model training
2:07:21 When an agent sends the wrong files
2:11:14 A loan application tests fair lending
2:16:09 Designing private benchmarks for enterprises
2:25:37 The leadership team on Handshake’s mission
2:28:36 When enterprises need their own models
2:33:03 The market for compute and expert data
2:39:12 Why labs hold out evaluation tasks
2:43:01 Hiring at Handshake AI