What Are We Trying to Augment?

I build AI harnesses, not the kind with straps.

The kind AI agents run inside: the tools they can call, the permissions they need, the loops they work in, the checks that stop them from doing something expensive on a Friday afternoon. My own products have AI inside them, and the part I keep returning to is rarely the model. It is the shape of the interaction around the model.

So this essay comes from an uncomfortable place. I design the very thing I am about to question.

Start with the question I keep coming back to: what if the measure of a good AI tool wasn't how much work it does for us, but how much more capable we become by using it?

What a new study found

In October, researchers presented a peer-reviewed paper, "AI Assistance Reduces Persistence and Hurts Independent Performance," at the Conference on Language Modeling. It was co-authored by a UC Berkeley researcher and covered by Berkeley News.

The team ran randomised controlled trials with 1,222 participants. In the first experiment, 354 people solved fraction problems. One group worked alone. The other had ChatGPT beside them, free to use for help or even to ask for answers.

The group with the AI did better while it was there. Then, after 12 problems, the researchers took the tool away. Almost immediately, that group stopped solving the questions. The group that had worked without AI kept going.

The pattern repeated with a larger group of 667 participants, and again in a third experiment with 201 people and an SAT reading comprehension task. When the AI support was withdrawn, both accuracy and persistence dropped. The people who had never had the tool were the ones who stayed with the task.

The paper gives a name to what the AI group lost: productive struggle. People made more mistakes, sure. The scarier finding is that they gave up faster.

I recognise the pattern in myself

I noticed this in my own work long before the paper gave me permission to notice it.

For years, my default move when something broke was to form a hypothesis. Read the stack trace, guess where the fault lives, open the file, and be disappointed or surprised. Lately I catch myself going straight to the agent instead. Paste the error, wait for the fix, move on. The answer arrives in seconds, and my hands never touch the reasoning that used to produce it.

Most weekends I still do things the slow way, on purpose. I keep one rule for myself: before asking, write down what I think is going on. It is a small ritual, and I cannot prove it works for me the way a trial proves it works for 1,222 people. But it keeps the muscle warm.

I use it every day, and my output has genuinely improved alongside it. What I object to is assuming that assistance and empowerment are the same thing. A system can make us more productive today without making us more capable tomorrow, and if we only measure the immediate result we will never see the difference.

The market rewards the wrong moment

Here is what actually bothers me: nobody in this story is behaving badly.

Technology companies compete. They need users who adopt, return and pay. In that world, speed and convenience are attractive measures of success. Judgement, curiosity and the ability to work a problem down to its roots are slower, quieter, and almost impossible to attribute to any single product interaction. What we measure is what we are selling. What we neglect becomes an externality.

Imagine two coding assistants. The first takes your ticket and returns a finished implementation, explaining nothing unless pushed. The second asks what you think is going on, shows you a different architecture you had not considered, reviews the design you proposed before it offers its own, and hands over more of the work as your competence grows. Both produce good code. The first optimises for the work. The second optimises for the work and for you.

Nobody ever asks for the second. No roadmap line rewards it. But I know which one leaves me sharper, because there was an evening I closed the session with the first assistant's fix merged and could not have defended a single line of it. We should ask why the second is not an explicit design goal anyway.

The harness is part of the product

I keep seeing interfaces described as if they were neutral. They are not. The interface decides what the model becomes in your practice.

A conventional harness asks whether the model solves the task, whether it does so reliably, whether latency and cost are falling, and whether more of the workflow can be automated. Those are my questions too. I ask them most weeks.

There is another set I should be asking:

Does the user understand what the system produced?
Is the system supporting their reasoning or replacing it?
Can they still do the task when the AI is unavailable?
Are they more autonomous after a month of using this than before?

These read like philosophy. They show up in code review, in design reviews, in how a team handles a failing build on a Friday. They matter in the domains where the answer to "can you redo that reasoning by yourself" has real consequences.

What a human-centred harness could look like

The goal is not to make AI worse at being helpful, and it is not to force a quiz after every prompt. It is to make assistance responsive to what the user is actually doing. Five principles worth designing for.

Think first, then consult. Before the full answer, offer the option to state a hypothesis. "What do you think is causing this? Share it and we'll dig in." The complete solution stays one request away. The difference is the moment of independent reasoning on the record.
Scaffold, don't substitute. A clarification, a hint, an explanation, a partial implementation the user completes. Not withholding. Matching. If I am learning recursion, the finished implementation defeats the purpose. If I am migrating a production service under a deadline, it is exactly what I need. A harness should tell those apart.
Make understanding observable. After generating a complex change, ask the developer to explain one critical decision. Offer a similar problem to solve without help. Optional, proportional to stakes, and never a performance.
Fade the assistance as competence grows. Detailed guidance early, shorter hints later, review-only eventually. The modes stay visible and adjustable, because the failure mode here is a tool that decides in secret how much help you are allowed.
Measure independence, not just assisted performance. Some products earn their keep precisely by removing the need for independence, and that is fine. Design for independence where the user values it and the task requires it.

Three modes, one afternoon

I can imagine a concrete architecture, because I have built primitive versions of it.

Explore: the user writes the first idea, the AI clarifies the question without taking over. Collaborate: the AI challenges assumptions, finds the missing information, offers another perspective. Delegate: the AI executes, with permissions and safeguards. None of the modes is the winner. The design win is making the switch explicit, with an intent check at the start: is the user trying to learn, to decide, to create, or just to get it done? The same person wants all three modes in one afternoon. I know I do.

Add progressive disclosure, a verification layer that separates established facts from assumptions, a feedback loop that occasionally asks for independent work, and an override that makes full assistance available without turning every interaction into a lesson. None of that needs a new foundation model. It needs an interaction layer, a small set of assistance policies, and the honesty to measure whether the policies actually help. The hard part is not building it. It is running the experiments.

The risks, stated plainly

An AI that coaches relentlessly becomes obstructive, so users must be able to pick their goals and switch the coaching off. A system that mistakes quiz compliance for understanding builds false confidence, so independent assessments have to be meaningful. People pressed for time, or working with accessibility needs, would be right to resent withheld answers, so context governs everything here. Questions designed to hold attention rather than develop skill would just be engagement with better manners. And a product that makes people genuinely independent risks losing some of the repeat usage that pays for it. Each of these is a real trade-off, none of them a reason to abandon the idea.

What are we trying to augment?

The word "augmentation" promises more than output. It promises capability: being able to reason an unfamiliar problem to the ground, to notice when an answer smells wrong, to persist when the solution is not arriving.

We have always used tools to extend ourselves, and delegation is often the smart move. Nobody should romanticise suffering through work a machine does better. The challenge is choosing what to delegate, what to preserve and what to develop. That choice is a design decision we make daily, whether or not we treat it as one.

The next generation of AI products could compete on intelligence and speed. They could also compete on something like cognitive fitness: how capable they leave people, how honestly they handle uncertainty, how much agency survives a year of daily use. Good assistance sometimes explains, sometimes challenges, sometimes steps aside and sometimes simply gets out of the way.

The promise I signed up for as an engineer was augmentation. I would like to build products that keep it.

Sources

#ai #software #engineering #design