Formally, the first group are "centaurs" (people assisted by machines) and the latter are "reverse-centaurs" (people conscripted into helping machines):

pluralistic.net/2025/05/27/ran…

Most workers have parts of their jobs they would happily automate away. I know a programmer who uses AI to take a first pass at CSS code for formatted output. This is a notoriously tedious chore. It's not hard to determine whether the AI got it right - just eyeball the output in a variety of browsers.

2/

If this was a chore you hated doing and someone gave you an effective tool to automate it, that would be cause for celebration. What's more, if you learned that this was only reliable for a subset of cases, you could confine your use of the AI to those cases.

Likewise, many workers dream of doing something through automation that is *so* expensive or labor-intensive that they can't possibly do it.

3/

I'm thinking here of the film editor who extolled the virtues to me of deepfaking the eyelines of every extra in a crowd scene, which lets them change the focus of the whole scene without reassembling a couple hundred extras, rebuilding the set, etc. This is a brand new capability that increases the creative flexibility of that worker, and no wonder they love it. It's good to be a centaur!

4/

Then there's the poor reverse-centaurs. These are workers whose bosses have saddled them with a literally impossible workload and handed them an AI tool. Maybe they've been ordered to use the tool, or maybe they've been ordered to complete the job (or else) by a boss who was suggestively waggling their eyebrows at the AI tool while giving the order.

5/

Think of the freelance writer whom Hearst tasked with singlehandedly producing an entire, 64-page "best-of" summer supplement, including multiple best-of lists, who was globally humiliated when his "best books of the summer" list was chock full of imaginary books that the AI "hallucinated":

404media.co/viral-ai-generated…

No one seriously believes that this guy could have written and fact-checked all that material by himself.

6/

Viral AI-Generated Summer Guide Printed by Chicago Sun-Times Was Made by Magazine Giant Hearst

The Chicago Sun-Times said "we understand this is unacceptable for us to distribute."

^{Jason Koebler (404 Media)}

Nominally, he was tasked with serving as the "human in the loop" who validated the AI's output. In reality, he was the AI's fall-guy, what Dan Davies calls an "accountability sink," who absorbed the blame for the inevitable errors that arise when an employer demands that a single human sign off on the products of an error-prone automated system that operates at machine speeds.

It's never fun to be a reverse centaur, but it's especially taxing to be a reverse centaur for an AI.

7/

AIs, after all, are statistical guessing programs that infer the most plausible next word based on the words that came before. Sometimes this goes badly and obviously awry, like when the AI tells you to put glue or gravel on your pizza. But more often, AI's errors are precisely, expensively calculated to blend in perfectly with the scenery.

8/

AIs are conservative. They can only output a version of the future that is predicted by the past, proceeding on a smooth, unbroken line from the way things were to the way they are presumed to be. But reality isn't smooth, it's lumpy and discontinuous.

Take the names of common code libraries: these follow naming conventions that make it easy to predict what a library for a given function will be, and to guess what a given library does based on its name.

9/

But humans are messy and reality is lumpy, so these conventions are imperfectly followed. All the text-parsing libraries for a programming language may look like this: docx.text.parsing; txt.text.parsing, md.text.parsing, except for one, which defies convention by being named text.odt.parsing. Maybe someone had a brainfart and misnamed the library. Maybe the library was developed independently of everyone else's libraries and later merged. Maybe Mercury is in retrograde.

10/

Whatever the reason, the world contains many of these imperfections.

Ask an LLM to write you some software and it will "hallucinate" (that is, extrapolate) libraries that don't exist, because it will assume that all text-parsing libraries follow the convention. It will assume that the library for parsing odt files is called "odt.text.parsing," and it will put a link to that nonexistent library in your code.

11/

This creates a vulnerability for AI-assisted code, called "slopsquatting," whereby an attacker predicts the names of libraries AIs are apt to hallucinate and creates libraries with those names, libraries that do what you would expect they'd do, but also inject malicious code into every program that incorporates them:

theregister.com/2025/04/12/ai_…

This is the hardest type of error to spot, because the AI is guessing the statistically most plausible name for the imaginary library.

12/

LLMs can't stop making up software dependencies and sabotaging everything

: Hallucinated package names fuel 'slopsquatting'

^{Thomas Claburn (The Register)}

It's like the AI is constructing a spot-the-difference image puzzle on super-hard mode, swapping the fork and knife in a diner's hands from left to right and vice-versa. You couldn't generate a harder-to-spot bug if you tried.

It's not like people are very good at supervising machines to begin with. "Automation blindness" is when you're asked to repeatedly examine the output of a generally correct machine for a long time, and somehow remain vigilant for its errors.

13/

Humans aren't really capable of remaining vigilant for things that don't ever happen - whatever attention and neuronal capacity you initially devote to this never-come eventuality is hijacked by the things that happen all the time. This is why the TSA is so fucking *amazing* at spotting water-bottles on X-rays, but consistently fails to spot the bombs and guns that red team testers smuggle into checkpoints.

14/

The median TSA screener spots a hundred water bottles a day, and is (statistically) never called upon to spot something genuinely dangerous to a flight. They have put in their 10,000 hours, and then some, on spotting water bottles, and approximately *zero* hours on spotting stuff that we really, really don't want to see on planes.

15/

So automation blindness is already going to be a problem for any "human in the loop," from a radiologist asked to sign off on an AI's interpretation of your chest X-ray to a low-paid overseas worker remote-monitoring your Waymo...to a programmer doing endless, high-speed code-review for a chatbot.

16/