Great news for xAI: Grok is now very good at answering questions about Baldur’s Gate

Different AI labs have different priorities. For example, OpenAI has traditionally focused on consumer users, while its competitor Anthropic tends to target enterprises. We recently discovered that Elon Musk’s xAI technology is specifically focused on video game walkthroughs.

On Friday, Business Insider’s Grace Kay published a detailed and far-reaching report on xAI, the artificial intelligence startup recently acquired by SpaceX, with a particular focus on how Musk is making life difficult for employees. But this particular story stood out:

In one case last year, the release of a model was delayed for several days because Musk was unhappy with the way a chatbot answered detailed questions about the video game “Baldur’s Gate,” according to people familiar with the matter. They said senior engineers were pulled from other projects to improve responses before launch.

Of course, you can imagine the frustration of any respected, experienced engineer who comes to work thinking he’ll address the fundamental problems of knowledge and machine intelligence, but goes out of his way to help a 54-year-old man beat his video game. But this anecdote raises a more pressing question: Did Musk end up getting the gaming skills he wanted?

To answer this question, Ram Iyer, our resident RPG fan, has put together a set of five trivia questions about Baldur’s Gate, which we confronted with xAI and the three main archetypes in a sort of semi-benchmark that I’ve decided to call Baldorpinch.

In the interest of journalistic transparency, I’ve published all the chat transcripts, so you can see them here: Grok, ChatGPT, Claude, and Gemini.

First, the good news: Grok actually provides very good information. Her responses were a bit dense with player-specific jargon — “scum save” instead of save and “DPS” instead of damage — but the answers were helpful and informed, provided you know what you’re talking about. Grok also really likes tables and theoretical crafts, which is about what you’d expect.

There are a lot of Baldur’s Gate guides, and the models were generally based on the same models, so the biggest differences were in style. ChatGPT prefers bulleted lists and sentence fragments, while Gemini likes that broad Important words.

The biggest surprise was Claude, who was particularly interested in giving me information that would spoil my experience in the game. When I asked about good pieces of music for parties, she concluded the directions by saying, “Don’t push too much and just play what sounds fun to you.” Thanks, Claude!

It’s important to keep in mind that this is a subject area that we know (thanks to Business Insider reporting) that xAI has specifically focused on achieving parity. So we shouldn’t read too much into the fact that after the announced race, Grok’s advice turned out to be the same as other models. However, it’s good to know that xAI can make it work if you try.