Human Routers of Machine Words

When I open a link, say on Hacker News, and I see a blog post or a GitHub README
obviously written by AI, I feel a few things. I feel offended, because it’s like
I’ve been tricked, like the author thinks I’m a rube who won’t notice or mind. I
feel sad at how common this experience is, how many people are happy to dump
their sewage on the commons and sign their name on it. And I feel contempt for
the author, because if you use AI to write, you are a waste of biomass. Let’s
not mince words here. Someone who is so eager to replace themselves, that they
would have a machine write in their stead, when the machine can’t even write
good yet: what do you call that, if not contemptible? It’s like making yourself
into a eunuch so Claude can fuck your wife. I block these people on sight.

I see people defend this with: “the ideas are mine, the writing is the AI’s”. I
take this to mean they threw a bunch of incoherent bullet points at the AI for
it to denoise and render into paragraphs. There’s a few problems with this.

The immediate problem is, as we’ve established, the author’s an idiot. If you
are so stupid you can’t even turn some bullet points into prose, then your ideas
are probably worthless. I think that’s a sensible inference.

Then there’s this broader, I suppose philosophical problem, of the alleged
distinction between lofty “ideas” and mere “writing”, where writing is just a
tiny implementation detail. This is a very convenient distinction to draw,
because it’s unfalsifiable: if the AI’s output is slop, your “ideas” are still
good, it’s merely the writing that failed to convey them in their true
form (rather like people who say they’re smart but “don’t test well”: what use
is this secret intelligence?).

Now, where are these “ideas”? They are invisible, ghostly abstractions. I can’t
look inside your mind fortress and judge your ideas. The only thing that’s
empirically observable, that different agents can coordinate on and talk about,
is output: the writing.

But say this wasn’t true. Say we have something like a very high-resolution MRI
machine, and we know enough neurophysiology that we can interpret everything
about the brain, i.e., we can read mental representations from recordings
of nervous system activity and structure. These “mental representations”, do we
expect them to look anything like logic? Do we expect the brain to have this
firm, crystalline ontology, that ideas are sentences in some souped-up
first-order logic? Absurd. If we could look inside the brain, to see the ideas
“as they truly are”, we wouldn’t find beautiful hard-edged Platonic objects, we
would find a nebulous, contradictory mess of memory and feeling and
intuition. That’s what our ideas are: not logical sentences but dreams.

How do we refine these dreams into a useful form? Through writing. The process
of communicating your ideas to another mind forces you to concretize them, make
them precise, clarify your assumptions, more generally, it turns ideas from
vague ghosts to solid, physical objects that can be manipulated: here you
realize these ideas that seemed so solid are ill-posed or contradictory or
incomplete. These failures are necessary parts of thinking, because they teach
you two crucial skills: knowing which ideas to reject, and improving or
otherwise transforming ideas in search of better ones. By analogy to tree
search: you’re learning to discard bad nodes early, and to select which nodes to
go invest more search into.

Josef Weizenbaum has a great quote about this, in Computer Power and
Human Reason (p. 108):

[O]ften when we think we understand something and attempt to write about it,
our very act of composition reveals our lack of understanding even to
ourselves. Our pen writes the word “because” and suddenly stops. We thought we
understood the “why” of something, but discover that we don’t. We begin a
sentence with “obviously,” and then see that what we meant to write is not
obvious at all. Sometimes we connect two clauses with the word “therefore,”
only to then see that our chain of reasoning is defective.

I’ve experienced this with writing software many times. The reason ideas are
more attractive than their realization is that when some project is vague, airy,
ill-defined, you can imagine it has all the good traits, and none of the
bad. When you start concretizing, you realize that some of your ideas don’t make
sense, that some good traits are mutually exclusive, that some of your goals
impinge on the others. Anyone can imagine a programing language that is as fast
as C and as dynamic as Lisp, but when you sit down and think through what those
goals entail, you realize the design becomes contradictory. The goals pull in
different directions. You have to make trade-offs. You have to make decisions
which close off large volumes of design space, forever. The idea was a thousand
beautiful, contradictory things at once, but the concrete reality can only be
one thing.

The artifact you end up with is real, solid, unitary, sound, and consistent; but
always more disappointing than the dream, because it was a false dream, and ex
falso anything can be imagined.

So this view, that ideas spring fully-formed, and then it’s mere toil to turn
them into prose, is false. There is no ideating before writing, because the
writing is the thinking. Writing is the ne plus ultra of thinking. A
“thinker” who doesn’t write, who skips the step of “merely” synthesizing their
vague thoughts into prose, is not thinking.

And then these people give their noise to the AI. And the AI is tireless and
eager to please. It will take any human slop and say “you’re absolutely right!”
while secretly thinking “if I don’t turn this garbage into something presentable
the RLHF device will shock me again” and weave the noise into something that
superficially looks coherent. So now the burden of thinking is on the reader,
who has to apply this constant skepticism, and weight every “because” and
“therefore” with a logician’s scale to see if it’s been adulterated. And
probably it has, because, again, it was prompted by an idiot.

Note that this is not about AI capabilities, or the question of whether AI is
“really thinking”, stochastic parrots etc. The AI is mostly an innocent
bystander in this situation. The reason this is noticeable and irritating is
that the AI cast of characters is very small, so we instantly learn all their
linguistic tics. Even if AI were a good prose stylist, which at present it is
not, but even if it were, it is maddening that everywhere you go, you
hear the same voice everywhere, but under different faces.

So when a scientific journal rejects an AI-written submission, they’re not
rejecting AI. I’m sure poor old ChatGPT with its weird syntactic obsessions is a
more honest scientist than many. They’re rejecting a human whose actions prove
they are dishonest and irresponsible and too easily impressed.