Does the human still make the decision?
A short piece on why it's not so simple
Imagine its your friend’s birthday in a couple weeks - they’ve just casually dropped it in conversation.
The Gold Standard
By your own volition when two weeks rolls around you “cared” enough to dutifully fight off the forces of forgetfulness. The birthday wish is gratefully received.
The 21st Century Compromise
Perhaps though like me your memory is bad. After all, you tell yourself, the ability to remember has little to do with how much you care. You put it in your calendar. Two weeks come by. You check your calendar and there it is! Your memory is jogged; you send the message; everyone is happy.1
The Agentic Era
You’ve heard all this buzz about OpenClaw. You forked out for a Mac Mini and now have a little AI assistant running 24/7. You haven’t really found a real use for it yet but you keep it running so you can say you have one. However! Now you connect the dots. You open up WhatsApp and text your AI immediately this birthday news you want to remember.
“Thank you for telling me benjosaur. What a wonderful thing to do for your friend. I have logged this information.”
You sleep blissfully having now discovered how AI can transform your life.
Two weeks pass.
“Good morning benjosaur. You are looking rather dashing today. By the way, it’s Ann’s birthday - you should wish her well.”
Thank goodness for AI. You promptly follow the advice and Ann is overjoyed.
“Wow Ben. How did you remember? I’m so pleased. You used to always forget.”
“Oh yeah haha it’s nothing really my AI Agent told me to do it.”
“You’re a psychopath.”
Wait. What just happened?
To (most) people this distinction seems pretty clear. While Ann loves tech, she did not sign up to be friends with an AI but here it is taking control of the decision to wish her well. This is wrong.
So far so good. Maybe we can use this “human owns the decision” principle as the north star when deciding whether automation is ethical. Indeed, when building in government, one of the many legal steers I have to follow is ensuring “decision making is not delegated” to AI.2
But is it really so simple?
I’m going to argue, to the dismay of senior stakeholders worldwide, that it’s not. And I promise the purpose is not just to rattle the cage of compliance that traps any early stage government project containing the word “AI”. The question is very important. But the way it’s often phrased presumes a much simpler world than the reality.
How different is the calendar and AI Agent really?
Imagine instead you hacked your calendar to hardcode a single instruction added to all notifications. The fated day comes and the birthday reminder now reads:
“Today is Ann’s birthday. You should message her.”
This feels wrong, almost in line with how the AI agent felt wrong. While functionally the notification is identical, the fact the 1s and 0s are now instructing you to do something crosses the line. It’s as if this extra code you yourself added has spawned a consciousness stealing all the credit of birthday remembering away from you. However, clearly the agency has always been driven by you - or at least past you.
Actually, this sort of thing already happens all the time. Even more, it wasn’t because past you coded any instruction, it’s just big tech’s quest to make your life frictionless. LinkedIn (pictured) is a particularly vapid example.3
Most have simply become accustomed to this extra instruction fluff, making it invisible. It doesn’t really change behaviour and so is just ignored.
Is it then inconsistent to bash the AI user?
Maybe it was too hasty to socially outcast the AI user. Perhaps it would have been more forgiveable and genuine if instead it was phrased as
“Oh yeah haha it’s nothing really my AI Agent reminded me.”
After all, the ultimate part of this decision making process was the final act of actually taking the effort to go and personally deliver the good wishes.
For life events at least, it is reasonably socially accepted to delegate the first parts of this decision, i.e. the remembering, to technology. Most aren’t freaked out either by LinkedIn taking the next step in deciding what to write by prefilling the congratulatory message. Perhaps this is because making the final call to send it is what matters. You were probably going to write something similar anyway.
Is this the answer then? Qualify the question as “Does the human make the final decision?”
The trouble is the scope of the final decision is determined by the decisions that came before it.
Going back to government, currently there’s a lot of buzz about automating labelling of public consultations. Here, the government asks what the public thinks about legislation X or policy Y, then (sometimes) many many people respond. One approach is to spend lots of money on reams of civil servants to manually go through them all. You might think the tech bro fantasy of “just do it with AI” is better; however, I would say this is not so obvious.
Determining the most important themes someone is talking about in their response is a normative insight which necessarily loses information. Often times it’s unclear what the right answer is and varying levels of inference can be used.
While it does feel like this is just the “process-heavy” end of policymaking, and so ripe for automation, the insights gleaned from this process informs the overall narrative and circumscribes the “final” policy decisions on what to legislate.
The automation of these earlier steps is then taking away part of the final decision. You can’t say it is still fully human, as different insights may have led to a different narrative, leading to a different scope of what policy responses are reasonable and so potentially a different conclusion.4
Unsurprisingly, more automation of the normative steps leading up to the final decision means less freedom with which the human makes their ultimate judgement. It is not possible to simply say humans still have all the decision making power.
Perhaps, if it were true that only non-normative steps were automated (e.g. whenever report X comes in put it into email Y) then it could be argued downstream human agency is not affected, as they themselves could not have done anything correctly different. However, the allure of LLMs is the promise of more flexible automation than possible before. This almost always means automating normative judgements. Take summarising. Multiple summaries can be “correct” but each with their own interpretive bend.
The Future
When ruled by that which is measurable, it is all too easy to see quantifiable cashable benefits of AI automation favoured over the unquantifiable loss of human agency.
This will happen gradually under the guise of moving humans to the ever more “empowered” and “strategic” parts of the production process.
I’m sure 20 years ago people would have been be markedly more disturbed by the idea of social media platforms prefilling messages for you. While you might freak out now if LinkedIn actually started just sending those messages, in 20 more years it’ll again be just “how it is” - the circus of social media travelling on.
“Ah well delegating remembering is fine the real decision is deciding what to write and actually messaging the person”
“Ahhh well delegating writing out the message is fine because the real decision is actually sending it”
“Ahhhhhhhh well delegating sending messages is fine because no real decisions happen on social media. Anyway why are you talking to me? Just put the UBI cheque in the bag bro.”
Before long we will have “empowered” ourselves into disempowerment; a royal family formality presiding over AI that actually governs.
The views expressed here are my own and do not represent those of my employer or any government department.
With thanks to Matthew Johnson for his helpful comments.
It’s curious how much of a knife’s edge the social acceptibility of friend tracking rests on. Putting their events in your calendar is cute but writing flashcards for spaced repetition is not. Writing about their key facts/tastes in your daily diary for a birthday gift is cute but collating that into a master doc on them is not.
This is more demanding than ensuring a human is accountable, else a solution could be “I don’t make the decisions but I’m happy to take the blame.”
It feels a pretty poor indictment by these apps on our empathetic ability as if without this guidance we would be powerless to interact with our friends.
A different final outcome does not necessarily imply evidence of a loss of human agency, since the fully human process is already noisy (different humans would draw different conclusions and conclude different policies). A more precise point would be that the final distribution of outcomes will be different when AI is involved. This is implied by lost human agency and not the converse.