GitHub – enjector/microgpt-c

A zero-dependency, pure C99 implementation of a GPT-style character-level language model.

The algorithm faithfully matches Andrej Karpathy’s microgpt.py — same architecture, same training loop, same sampling — but compiles to native code with optional compiler-driven SIMD auto-vectorisation for dramatically faster training and inference.

Train a GPT in 20 ms. Generate names in microseconds. No Python. No PyTorch. No GPU.

MicroGPT-C is a minimal, readable implementation of a GPT (Generative Pre-trained Transformer) — the same family of models behind ChatGPT, but stripped down to its essential algorithm. It trains a tiny character-level language model that learns to generate realistic human names from scratch.

The goal is education and experimentation: understand how attention, backpropagation, and the Adam optimiser actually work at the lowest level, without any framework abstractions.

# Linux / macOS
chmod +x build.sh
./build.sh
./build/microgpt

:: Windows
build.bat
build\Release\microgpt.exe

The build automatically copies data/names.txt next to the executable.

Measured on the same workload (1,000 training steps, 20 inference samples) — C vs the reference Python:

INT8 quantised build: ~25% slower training than fp64 on this tiny model, but ~8× smaller weight storage — ideal for constrained devices.

A single-layer, decoder-only Transformer following the GPT-2 design:

Input → Token Embed + Pos Embed → RMSNorm
  → Self-Attention (4 heads, causal) → Residual
  → RMSNorm → MLP (fc1 → ReLU → fc2, 4× width) → Residual
  → Linear (lm_head) → Softmax → next-token probabilities

Training uses the Adam optimiser with linear learning-rate decay (configurable in microgpt.h).

The --simd flag enables compiler-driven auto-vectorisation of the core dot products, matrix multiplications, and normalisations. On x86-64 the compiler targets the best available instruction set (SSE4, AVX2, etc.) via -march=native; on MSVC it enables /arch:AVX2. This gives a measurable speed-up on larger models without any hand-written intrinsics — the compiler re-writes the scalar loops into SIMD instructions automatically.

# Linux / macOS — auto-detect best ISA
./build.sh --simd

# CMake directly
cmake -DMICROGPT_SIMD=ON ..
cmake --build . --config Release

Weights are stored as 8-bit integers with per-matrix scales — the forward pass dequantises on the fly; Adam updates an fp64 master copy and requantises each step. This reduces weight storage by ~8× (37 KB → 4.6 KB) at a small accuracy/speed trade-off.

mkdir build && cd build
cmake ..
cmake --build . --config Release

# With INT8 quantisation
cmake -DQUANTIZATION_INT8=ON ..

# With SIMD auto-vectorisation
cmake -DMICROGPT_SIMD=ON ..

# Both
cmake -DQUANTIZATION_INT8=ON -DMICROGPT_SIMD=ON ..

microgpt_amalgamated.c is a self-contained single file containing the full GPT algorithm — data loading, training, and inference. No header file needed:

# Compile directly (no CMake required)
cc -O2 -o microgpt microgpt_amalgamated.c -lm
cp data/names.txt . && ./microgpt

# Or via CMake
cmake --build build --config Release --target microgpt_amalgamated
./build/microgpt_amalgamated

• C99 compiler (GCC, Clang, MSVC)
• CMake 3.10+
• No other dependencies

MIT — see LICENSE and source file headers.

Author: Ajay Soni (ajay.soni@enjector.com), Enjector Software Ltd.