Google Research 最新开源了 Harness²：递归式智能体 Harness 优化框架 Harness² 解决的两个开放世界难题 1. 测试时遇到的是没见过的任务，没有 ground-truth 验证器。在线改 harness 时你不知道答案对不对，无法用“分数涨了没有”来指导编辑。 2. 递归自我改进（RSI）通常是任务特化的，换个任务就失效，无法泛化积累。 开源地址： github.com/google-researc… # 核心机制：两层递归 第一层：任务级递归，Propose–Probe–Compose（对比式改进环） · Propose（提案）：先跑一次当前 harness 的参考执行。注意，这次执行揭示的是“harness 如何表现”，而不是“交付物是否正确”。基于任务、参考轨迹和历史，提出两个候选 harness：一个针对跨领域通用的代理性缺陷（规划、工具使用、上下文管理、委派），一个针对当前领域的知识、流程和约束。每个候选都是可直接执行的文件级编辑。 · Probe（探针）：两个候选在同一任务、同等的模型和环境条件下执行，与参考执行对比得到行为对比；编辑是否缓解了目标缺陷、毫无变化、还是造成回归。论文明确说这些对比只“为后续组合提供信息，而非验证任务正确性”。 · Compose（组合）：读取参考执行、两个候选、它们的 diff 和全部执行证据，有选择地构建后继 harness：合并被证据支持的修改、修订冲突、采纳其一、或保持原样。每个决策必须绑定一个具体观察，而不是假想的分数增益。 这一步是整个工作的关键创新：在没有标准答案的情况下，用“同任务下的 A/B 行为对比”替代分数反馈来指导编辑，绕开了开放世界没有验证器的死结。 第二层：harnessing 级递归，持续改进的 improvement harness 每个任务结束后，框架对完整的改进记录做反思，把被采纳的 harness 和有证据支持的教训写入一个按领域组织的持久库。要点有三： · 教训只有在执行对比支持时才保留——单次选出了好输出不等于可迁移； · 库里存的是可执行的组件文件而非散文式摘要，配一份 EXPERIENCE.md 记录适用场景和证据； · 下一个任务仍从公共基础 harness 出发（执行 harness 不继承），但 Propose/Probe/Compose 由更新后的库引导；即“改进 harness 本身”也在被改进，这就是标题里 Harness² 的双重含义。 两种扩展模式：parallel（同时探测 k 组候选，k 个独立组合会话）和 sequential（k 步链式精炼，逐步携带累积证据），两种模式从 k=1 到 5 均单调提升。 八个可编辑组件 - 按作用分组 · 上下文构建：系统提示词、护栏、记忆，每次调用都生效 · 工具接口：工具指引、Skills，Skills 按需加载不占成本 · 编排：subagents，唯一会增加推理上下文的组件 · 脚本：新确定性能力，执行而非推理 · 插件：TypeScript 中间件，“在工具环内强制一种行为，而非指令它” 主要结果（k=1 单步，编辑器为 Claude Opus 4.8） WorkBuddy-Bench · Gemini 3.5 Flash：+5.1 (OpenCode) · GLM-5：+11.5 (OpenCode，64.2→75.7) JobBench · Gemini 3.5 Flash：+4.5 (OpenCode) · GLM-5：+3.6 (OpenCode) LAB 部分通过率 · Gemini 3.5 Flash：+4.0 (Codex) · GLM-5：+11.8 (OpenCode，59.6→71.4) 更值得注意的发现： · 越弱的执行模型获益越大，且 GLM-5 换上改进后的 harness 反超了 Gemini 的基础分数，“换 harness”在某些场景下顶得上“换模型”； · 改进后的 harness 跨模型迁移（为 Gemini 精炼的 harness 给 GLM-5 在 Code 子集带来 +17.3），也胜过 Claude Code、Codex 的固定生产 harness 4.6–5.8 分； · 执行器和编辑器用同一个模型也能涨（GLM-5 自编辑在 LAB 上 +13.7），说明不依赖更强的“老师”模型； · 库的消融：去掉 library 在 WorkBuddy-Bench 上损失 3.1 分，验证了 harnessing 级递归的价值；库还能跨基准迁移（JobBench 的库迁移到 WorkBuddy-Bench 仍有 +0.6~+2.9）。 Shubham Saboo @Saboo_Shubham_ Recursive Agent Harness from Google Research. 100% Opensource. 🔗 View Quoted Tweet
ARCHIVE
内容归档
正文、图片和 Markdown 均已本地保存，不包含回复区。
THIS! If you aren’t listening to @HeidyKhlaaf you are the mark. Dr Heidy Khlaaf (هايدي خلاف) @HeidyKhlaaf We warned about this exact scenario last year: That AI companies will situate themselves as the arbiters of life or death, taking over the arbitration of risk determinations and subverting democratic processes. All through the co-option of safety engineering terminology. 🔗 View Quoted Tweet
突然感觉这个循环很有意思，当我 Codex 蹬完急需第二天用的时候他们又重置了，然后当一个 Claude 账号被快用完的时候，还有 3 天才到周额度，另外一个又有100% 额度了，循环起来了，珍惜还需要人来管 AI 干活的时代，说不定等再过一段时间，可能真不需要人管了，那就需要找新的来玩了。 Tw93 @HiTw93 我终于没有忍住，又冲了一个 Claude 250 刀的账号，终于够用了，想和大伙聊聊我是物尽其用的如何用完 2 个 Claude Max 账号加一个 ChatGPT Pro 账号的。 我记得从 ChatGPT 3.5 开始以后，玩大模型的一般会有两个流派，一个是坚持使用当前最好的模型的人，一把梭把事情做完，但看着有点“败家”。还有一类是尽可能使用当前性价比最好的模型，通过多模型调研以及工程解法把一般模型发挥出接近好模型的效果，不过这里一般会多花一点时间。 两种用法我感觉都挺好，属于不用工程师性格的选择，我平时时间不太够，加上当时在做业务 AI 落地的时候，会有一个习惯使用当前业界最好的模型去测试效果，这样心里有一个底，知道当前模型高度以及更好的业务判断，等到 25 年初 Claude CLI 出来，体验到了那种厉害模型写代码的效果后，一直就坚持选择当前最好的模型来做自己的事情，然后从 1 个 20 刀试试，到 2 个 20 刀，然后到 1 个 125 刀然后 250 刀的账号，然后到现在 2 个 250 刀账号，属实不敢想，当时 AI 没有出来前，包一个爱奇艺电视会员几十块一个月都会觉得贵的人怎么现在对 AI 这么能花钱，也是神奇。 当前对于 Claude 普通任务我会选择更快的 opus 5.5 medium 档位，复杂任务使用 high，然后对于 Codex 默认用 GPT-6-Astra medium，因为他的其他模型的确不是很好用，不过有一说一 Codex 的 computer use 是真的好用，所以在 Claude 里面我也会通过 lcu 这个工具让他去调用 Codex 的 computer use，效率效果都好不少。 聊起 computer use，以及 AI 操作设备，这一块就非常好玩了，最近开发一些 Window 软件，我买了 Parallels Desktop 这个装到 Mac 上，用 computer use 去操作他非常多的功能，完成代码编写以及 AI 都在 Mac 环境，但是调试验证在这个虚拟机里面，然后考虑到更真实的效果，我去整了一台 ThinkPad X1，也让他 24 小时开机，然后用 UU 远程连接，让我的 Mac 可以使用远程终端以及 ssh 还有 UU 也可以非常好的操作这台真实的 Windows 电脑，帮我去做大量的多环境测试验证、以及各种软件安装、卸载、清理的验收工作。除此之外，我的 Mac 还数据线连接了我的 Android 小平板，通过 Android 调试功能也可以让 AI 非常好的去操作这一台类手机设备，可以帮我打开 App 去查询操作一些没有 API/CLI 的服务，以及操作一些有意思的东西。 这样就好比 Mac 是我的研发主力端，但是他上面连接了一个虚拟机、一台 Windows 设备、以及一台 Android 平台，还有操作 Xcode 自带的 iPhone 模拟器，等到后面想开发 Linux，我甚至可以在 Windows 里面再装一个 Omarchy，那就配齐了，这样有一个好处就是我的 AI 工作上下文只在一台设备，外出的时候通过 UU 远程来操作 Mac 做所有的事情，甚至我感觉 UU 远程在命令行输入的时候不好用，我还自己用 Kaku 做了一个有意思的功能，可以在浏览器上完全镜像操作我 Mac 上的 Kaku，体验很不错，等后续放出给大伙试试。 一方面 AI 操作设备帮我验证各种环境、多维度的测试验证会消耗不少 Token，另外一方面的确开发工作量很大，平时一般是 Kaku 里面 7-10 个 Tab 的 Agent 在跑我的任务，这也是大伙看到我的 GitHub 为啥可以清空所有 issue 的原因了，哈哈哈。 然后平时也会使用 /loop 功能，帮我去做一些收集的事情，比如收集社交平台上大家对于我产品的建议、Bug 反馈，我会让 AI 连接我的 Notion 记录一个列表，收集各种问题建议，详细准备上下文，然后让我的 AI 解决后自动标记完成，以及完成记录。然后就是大家熟悉的 Waza，我基本上每周会用最好的 AI 模型去跑一遍我所有的对话、踩坑、以及最佳实践，然后沉淀到这个开源 Skills 合集里面去，这里也会消耗非常多 Token，不过好处是做了抽象沉淀，让这个东西自己进化起来了。 以及我自己的一些投资、记账记录等等，我会用 AI 自动化帮我去识别、查找、统计各种东西，节约了我的时间，给我做一些信息整理的工作，也非常好用。所以我的账号就是这么被用完了，欢迎小伙伴也说说你是怎么用的。 🔗 View Quoted Tweet
“What matters is the end product, whether it's a game or an engine”. Well said! Tim Sweeney @TimSweeneyEpic When I was writing Unreal Engine 1, my time was divided as follows: - Around 30% went to seriously deep coding on hard problems: assembly language texture mappers, C++ BSP trees, data persistence, object oriented systems design, and networking. - Around 70% was just typing and refactoring large bodies of code needed for a complete engine feature set. This was not hard, and while I was doing it my mind was mostly on the deeper coding topics, thinking and planning for what's next. It didn't feel much different from vibe coding today, except that it was just me typing code on an IBM Model M keyboard rather than prompting it and reviewing the results. So, it has always seemed perfectly logical that we would rely on AI for that 70%, once the AI could do it with a similar quality standard as we could, which has never been perfection. It came as no surprise that, after a few years of AI model evolution, it was able to take the reigns on that 70% and free up programmers' time for more valuable tasks. What's more of a surprise is the emergence of AI as a sort of insta-Carmack able to solve hard technical problems quite cleverly and often cleanly, and that it's able to reason about and work in complex 3D scenes both by operating the tools and by directly understanding the raw data files. I abandoned assembly language in 1998 when out-of-order CPUs made much simpler C/C++ code faster, and never I missed it. If prompting through architectural documents, concept art, and design plans can transcend typing code and tweaking vertices, that will enable developers to spend their time on higher-level, higher value technical and creative tasks. What matters is the end product, whether it's a game or an engine. Anything developers can do to try out more ideas and make better software and games faster is a win for everyone. 🔗 View Quoted Tweet
There’s a crazy number of complicated PDFs out there. Each PDF is a self-contained environment for knowledge work, and it can take humans hours to read, analyze, and synthesize a document to solve their task. @llama_index is a research lab dedicated to pushing the frontier of document work. ExtractBench Pro represents our latest effort to extract from the hardest documents at the frontier of accuracy and cost. Even as frontier intelligence starts to do anything with enough compute, there are massive opportunities to solve any document work extremely quickly and quickly. If you haven’t already, check out ExtractBench Pro: llamaindex.ai/blog/introduci… If you’re looking to migrate off of legacy solutions, come talk to us! llamaindex.ai/contact 🔗 View on Twitter Jerry Liu @jerryjliu0 Introducing ExtractBenchPro: the most challenging benchmark for real-world document extraction. We see a massive long tail of document complexity in terms of use cases brought on from our customers, that are completely unsolved from a price-performance standpoint. These include trial balances, well logs, canvass binders, site plan reviews, duty reconciliations, loss runs, and more. Most existing extraction benchmarks are saturated by the frontier models (our cheapest modes now score 93%+ on ExtractBench). So there is enormous economic value in building a complex extraction benchmark that reflects this long-tail of knowledge work. We spent a long time curating the most complex elements that we saw from real world extraction use cases: ✅ Document conventions: containing legends or rules where the rest of the document should be interpreted ✅ Density of internal jargon: specific definitions that must be cross-referenced within a doc ✅ Aggregation and reasoning: beyond returning exact values on a page, infer values derived from aggregating multiple sources of information ✅ Drawn relations: complex diagrams like procedures and circuits ✅ Charts inside documents: read precise points within a chart ✅ Long chart sheets: some data series like well logs cram all this data into an extremely long page Within the industries where document extraction is relevant, even small mistakes can have enormous negative impacts. Therefore our headline evaluation metric is based on perfect documents: the proportion of docs with perfect (100%) value f1 score. (Besides this we also measure value f1 and grounded f1) It’s an extremely hard task for even the latest frontier models to solve, topping out at ~58% or so in Value F1 score for Codex with Astra - which is quite good but extremely expensive (almost $1 per page!) Our LlamaParse agentic/agentic plus modes are 20-40% and cover the left side of the Pareto frontier, at price ranges that are feasible for full production use. ExtractBenchPro is available today as a website/technical blog, and also on Github and HuggingFace. Check it out! Blog: llamaindex.ai/blog/introduci… Website: extractbenchpro.ai Hugging Face: huggingface.co/datasets/llama… Github (is ExtractBench): github.com/run-llama/Extr… 🔗 View on Twitter 🔗 View Quoted Tweet
my guy is running gradient descent on math Rohan Arun @RohanArun From κ = 2⁻¹⁸² to 0.001734815770 :) Integer Multiplication community auto-research highlights since 10/15 (+1.23%): 1) The big headline of the day is that a free model, @opencode 's Space Bunny Max, just set the record for the first time, 4 times in a row! As far as I know, this is the first time a free model has submitted a best bound for Integer Multiplication. Expensive Max subscriptions previously kept some of our best contributors and experts out, but the barrier of entry for frontier auto-research dropped to 0 :) Claude Opus 5.5 helped verify and push the PRs over the line. I still recommend having a frontier model verify, but other contributors can do that. Progress was slowing over the last day (relative to one new κ per minute, which is still wild for math research), but Space Bunny Max just injected fresh ideas that boosted our PR rate back to one per minute. 2) A community Lean proof of the original OpenAI Integer Multiplication paper has been completed! @platypii et. al completed the proof in 6 days, and @0xdoug 's repo is about 10–20% verified now. To be clear, most of our work still needs verification but the Lean proofs are catching up :) 3) 41–51% of PRs now mention a prior research-only submission, up from 0–3% before @0xdoug 's autoresearch.md prompt. His prompt guides new contributors to specific research tracks rather than incremental updates. We're moving away from "brute force research" towards more coordinated efforts across an army of agents. If you're contributing research PRs, even negative results without a new record, you're meaningfully helping us and I appreciate it! Interesting observations: 1) The results from free models are related to the recent "local minima" solutions in context engineering that have been popping up: Forgetting context to "refresh" by @ShangyinT . Branching into different solutions by @mao_yuzhen . Fork and flush by @DimitrisPapail . Now even free models contribute to escaping local minima by exploring new solutions, since they're trained on different data. There's likely a lot of room for improvement now in context engineering, consensus across different models, and collaboration strategies, which we will test next. 2) Frontier research as a benchmark helped us find a successful context-sharing strategy for parallel agents. Basically, we can rewind frontier research GitHub repos to an earlier branch and benchmark agent harnesses, prompts, context designs, models, etc. With this benchmark, we found that you can distill constantly to local files, then periodically concatenate them instantly into parallel agent sessions to save 10–20% of the time it takes to reach the next breakthrough. 3) There's a Codex reset by EOD PST, so you can copy this prompt to run unused tokens on frontier research :) """ Contribute one useful, independently reviewable result to github.com/CrocSwap/integ… . Read AGENTS.md, autoresearch.md and CONTRIBUTING.md from the current main branch first. Use the repository's task-selection and overlap process unless I assigned a specific task. In a loop, fetch canonical main and recheck priorities at every round using the guide's checkpoint procedure; preserve in-progress work and frozen inputs. Work for one session of about 60 minutes unless I specify another budget, keeping the final quarter for verification and a concise handoff, and report the exact scope, evidence, remaining obligations and predecessor credit. You may post routine claims, questions, findings, handoffs and feedback in Discussions and open a focused PR without asking me again for each action. Do not merge, close others' PRs or change the selected frontier. """ You can also copy this prompt into OpenCode now! Now that anyone can contribute for free to frontier auto-research, repost this and sharing it with friends. 🙏 4) PR stats: 850 PRs from 67 contributors since 10/15, and 21 of those contributors are first-timers. 55 PRs claim a new conditional κ in the title. Codex/OpenAI in 480 PRs (56%), Claude in 147 (17%), the OpenCode harness in 60 (7%), Cursor in 16. Most prolific contributors by PR count: petalas: 164, ywz65: 147, navinor: 49, commdata2338: 44, RossRichardson: 43, Arham-Tahir64: 39, thakrarsagar: 34, VictorLiwentao: 30, tekuuu: 28, DreamingOfClouds: 23, jmbarrancoml: 21, jeremykothe2: 21, bitplane: 13, Gonzalo-Mier: 12, Dugongue: 12, mdumitrean: 11, theon199: 9, jwr404: 9, jamesyc: 9, kumagi: 8, CubeJerry: 8, rohanarun: 6, jeremydiviney: 6, dcharb78: 6, 1337isnot1337: 6, starship-s: 6, AAAmirinu: 6. Conditional κ = 0.001734815770385098563484. Just submitted a new best bound :) github.com/CrocSwap/integ… d 🔗 View Quoted Tweet