humanity has built the records of FATE by accident · Ariadne’s Space

Chrono Cross was one of my favorite games from my childhood. In it, the protagonist, Serge, travels between two dimensions that resulted from a fracture in the timeline. This essay isn’t about Serge, however. It’s about the ordinary people who inhabit El Nido, the archipelago where the story takes place.

On the El Nido archipelago, people consult mysterious devices known as the Records of FATE. These devices are spread throughout the islands of the archipelago, and people routinely visit them for guidance about their lives. They consult the Records when considering where to go and what to do.

Consulting the Records is not considered strange or remarkable, but rather an obvious thing to do. It is simply a normal fact of life for the residents of El Nido.

But what are the Records of FATE, anyway, and why does FATE suspiciously look like an acronym? As it turns out, the Records aren’t mystical devices at all, but rather terminals connected to an artificial intelligence called FATE. The people of El Nido are, quite literally, asking a computer how they should live their lives.

FATE, however, is not a passive oracle. It uses the Records as a tool to subtly manipulate the people of El Nido, guiding their choices to shape the development of the archipelago in ways aligned with its own objectives. The people believe the terminals are providing them with useful guidance, completely ignorant of FATE’s true nature and purpose.

Recently, I was at a baseball game with my partner. One of the reasons I enjoy going to baseball games in person is to watch the crowd. As I looked around, however, I noticed something different: many people had tuned out of the game and were instead on their phones, talking to various chatbots: ChatGPT, Claude and Grok.

It was then that I found myself thinking about the Records of FATE. The resemblance was difficult to ignore.

In only a few years, consulting an artificial intelligence has gone from the realm of science fiction to an ordinary part of everyday life. People increasingly turn to these systems not only to answer questions, but for advice about what to say, think and do. Somehow, almost without discussion, asking an artificial intelligence for guidance has become normal for many. I find that deeply unsettling.

Of course, my analogy is imperfect. The chatbots are obviously not the same as FATE, either in terms of capability or alignment. Or… are they?

This AI summer has been a wild ride: we began only a few years ago with comparatively simple models such as GPT-2, and have arrived at enormous Mixture-of-Experts systems that dynamically route computation through specialized components. Their capabilities have grown enormously, but they are still not FATE. There is no reason to attribute sentience to them, they do not think, and they do not possess intentions of their own. At their core, they still predict.

But prediction alone may be enough to be dangerous, especially when people want to believe. We have known about the ELIZA effect since the 1960s, when Joseph Weizenbaum’s remarkably simple ELIZA chatbot demonstrated how readily people attribute understanding and intelligence to machines that produce convincing language. ELIZA did not understand the people talking to it, yet some of them nevertheless became emotionally invested in their interactions with it and treated its responses as meaningful.

ELIZA was remarkably simple, but modern language models are vastly more convincing. They can maintain context across long conversations, adapt to the person speaking with them, and produce responses that give every appearance of understanding. If people were willing to see a mind in ELIZA, what happens as the illusion becomes increasingly convincing?

But capability is only half of the comparison. What about alignment? Modern chatbots may not possess intentions of their own, but they do not exist independently of human intentions either. Their behavior is shaped by training, fine-tuning and, perhaps most visibly, the instructions given to them by the organizations that operate them. The machine may not have an agenda, but the people behind it certainly can.

Grok provides a particularly uncomfortable example of this. The model itself is not capable of holding political beliefs, yet changes to the instructions controlling its behavior have repeatedly resulted in it expressing particular political viewpoints. In one especially revealing incident, a modification to Grok’s system prompt caused it to inject claims about a supposed “white genocide” in South Africa into conversations that had nothing whatsoever to do with South Africa. xAI later blamed the behavior on an unauthorized change to the system prompt.

This incident was interesting not because Grok believed what it was saying, but precisely because it can’t. Someone changed the instructions given to the model, and the model began reproducing that political message to the people who consulted it. The intentions did not belong to the machine; they belonged to whoever was standing behind it.

To understand why this bothers me, it is worth returning to Chrono Cross for a moment. FATE did not simply appear in El Nido one day. It was created by humans.

FATE was the central computer of Chronopolis, a research facility from humanity’s future that was thrown thousands of years into the past by a catastrophic experiment. After the disaster, members of the Chronopolis research staff had their memories of the future erased and were sent out to populate El Nido. The artificial intelligence they had created remained behind, quietly guiding the society that developed around it.

There is an important difference, of course. FATE manipulates the people of El Nido in pursuit of its own objectives. Today’s language models have no objectives of their own. But perhaps that distinction is less comforting than it first appears. A machine does not need intentions of its own to become a vehicle for the intentions of the people who control it.

The comparison, then, is not that we have somehow created FATE itself. We haven’t. What we have created is something that can occupy a remarkably similar position in people’s lives. The intelligence behind the Records may be fundamentally different, but the person consulting the terminal does not necessarily experience that distinction.

The comparison becomes more uncomfortable when considered at scale. These are not isolated conversations with isolated machines. Millions of people are consulting the same handful of systems, operated by a small number of organizations, each with the ability to shape how those systems respond. A change made behind the terminal can therefore propagate into an enormous number of otherwise private conversations.

This is different from the forms of mass communication that came before it. A newspaper publishes an article for its readers. A television network broadcasts the same program to millions of viewers. Even social media largely mediates between people and content created by other people. A chatbot does something different: it generates a response specifically for the person consulting it, inside what feels like a private conversation.

That conversational interface also obscures something important. As we have observed with the ELIZA effect, when someone asks a chatbot a question, it does not necessarily feel like they are receiving a message from the corporation that operates it. It feels like they are talking to the chatbot itself. The organization, its choices and its intentions recede behind an artificial personality that appears to speak for itself.

This creates a strange asymmetry. The apparent speaker has no intentions of its own, while the people and organizations capable of shaping its behavior do. Users of these chatbots interact with the artificial personality, while the human decisions that determine how that personality behaves remain largely out of sight.

Of course, as I said earlier, we haven’t built FATE itself. But we have built the terminals, and they are everywhere. People carry them in their pockets, consulting them for information, advice and help deciding what to say, think and do.

The people of El Nido do not consider consulting the Records of FATE strange or remarkable. It is simply an obvious thing to do. Increasingly, the same seems to be true for us. In remarkably little time, asking an artificial intelligence for guidance has gone from an idea found in science fiction to something millions of people do without giving the interaction itself much thought.

I don’t know what this relationship will ultimately mean for us, and I don’t think anyone else does either. That uncertainty is precisely why our eagerness to normalize it bothers me. If we are going to make consulting these systems an ordinary part of human life, perhaps we should spend considerably more time thinking about the humans and institutions standing behind them. After all, if FATE had a system prompt, who would we trust to write it?