RightSignal · Friday, October 16

AI agents struggle with real customer projects where requirements are unclear

Dongxu Yang, Hongda Yan, Renxin Wang et al. · Submitted Oct 15 · 227 upvotes yesterday

Preprint · not peer reviewedTrending

The authors built 95 delivery tasks from real customer projects across 18 sectors. Agents must talk to simulated stakeholders, find unstated constraints and change live-like systems. 13 model setups from seven providers were tested.

Key finding: The best setup fully solved only 14 of 95 tasks. Finding unstated requirements was the main obstacle.

Why it matters: Most agent tests give clear instructions. Real work rarely does, so this measures a gap that matters for companies deploying agents.

Read on arXiv

An open toolkit for AI agents that operate computers by clicking and writing code

Hanzhang Zhou, Long Li, Quyu Kong et al. · Submitted Oct 15 · 265 upvotes yesterday

Preprint · not peer reviewedCode releasedTrending

OpenGCUA is an open project for agents that use a computer through both the screen and code. Letting the agent run code means it can use results to decide its next step.

Key finding: On the OSWorld-v2 test, it raised scores for all eight models tried and cut steps per task by 53% on average.

Why it matters: The best computer-use systems are closed. An open version lets others study and improve them.

Read on arXiv

New video test shows AI models still struggle to say when and where things happen

Xuyang Liu, Shihao Wang, Xinyu Wei et al. · Submitted Oct 13 · 565 upvotes yesterday

Preprint · not peer reviewedTrending

OmniCompass is a hand-checked benchmark of 820 videos that asks models when something appears, where it is and what happens, with a track that also needs the audio.

Key finding: Across 50 models, open models trail closed ones, especially on sound plus video, and precise timing and location stay hard for all.

Why it matters: It shows where video AI is weak before people rely on it for tasks like reviewing footage.

Read on arXiv

A smarter way for small AI models to learn from a bigger model's training

Bingxi Hou, Guochao Jiang, Guofeng Quan et al. · Submitted Oct 15 · 450 upvotes yesterday

Preprint · not peer reviewedTrending

When copying what a teacher model learned, the method checks a midway snapshot of the teacher. It only passes on changes that pointed the same way early and late in training.

Key finding: It scored best among the compared methods in all six maths and coding settings, at the cost of one extra scoring pass.

Why it matters: Cheaper, smaller models could pick up reasoning skills more reliably from expensive ones.

Read on arXiv

AI safety

What happens when AI agents rewrite their own tools and instructions

Chuyang Wei, Haoxiang Guan, Minghang Zhu et al. · Submitted Oct 13 · 185 upvotes yesterday

Preprint · not peer reviewedCode releasedTrending

A 66-page framework for agents that keep revising their prompts, memory, workflows and code while the model stays fixed. It includes rules for accepting a change only after checking it on fresh data.

Key finding: The authors argue reliable use needs ongoing checks and separate safeguards on what the agent is allowed to do.

Why it matters: Self-improving agents are moving into real use, and this sets out how to keep their changes in check.

Read on arXiv

Your RightSignal

Topics in News

AI agents struggle with real customer projects where requirements are unclear

An open toolkit for AI agents that operate computers by clicking and writing code

New video test shows AI models still struggle to say when and where things happen

A smarter way for small AI models to learn from a bigger model's training

What happens when AI agents rewrite their own tools and instructions

That’s every paper.