A Rough Idea LLM Explorables/Try a Model as It Improves
How a Model Gets Made

Try a Model as It Improves

How might you experience a model improving as it is trained? Within a single ten-minute run, what changes between where a model starts and where it ends up? As its training script evolves across a session, what changes in how it responds to prompts? And given the same prompt, how do models trained on two different datasets respond? This explorable gives you access to the tools and infrastructure to explore these questions for two small datasets.

Within a single ten-minute run, what changes between where a model starts and where it ends up?

Same: the model, its training script, the prompt. Changed: how long it had trained.

Dataset

Pick two moments. Ringed ones answer your own messages.

The two datasets

Every model on this page learned from one of two small datasets of short stories, different in kind, size and source. Each run trains a new model on one of them for ten minutes.

TinyStories

Short stories for young children, written by GPT-4 using only words a typical three-to-four-year-old understands.

From Made in 2023 by Ronen Eldan and Yuanzhi Li of Microsoft Research for TinyStories, a paper asking how small a model can be and still produce coherent English. These models use Andrej Karpathy’s cleaned copy of the GPT-4 stories from the original dataset.

Why it is here It is the dataset Karpathy’s autoresearch starts from, and the paper’s finding is this page’s premise: a model this small can learn to produce coherent English from it.

Size
2.7 million stories · 2.2 billion characters

Folktales

Folk and fairy tales and myths in English, from the public domain.

From A public-domain collection on Hugging Face, compiled from D. L. Ashliman’s Folklore and Mythology Electronic Texts at the University of Pittsburgh and a Kaggle set of fairy tales.

Why it is here The contrast: a different kind of story, in older language, and tiny, about 180 times smaller than TinyStories. A ten-minute run goes through it several times over, so its models can start to copy it.

Size
12 million characters

Their scores can’t be compared: each measures how well a model predicts its own dataset’s text, kept out of training.

What you should know now

How might you experience a model improving as it is trained? By giving the same prompt to two versions of it and reading what each writes.

  1. Models reflect datasets. Given the same prompt, the Folktales models write folk tales and the TinyStories models write stories for small children.
  2. Models improve with training. In a single ten-minute run, a model goes from random pieces of its vocabulary to words, then phrases, then the opening of a story.
  3. Model training algorithms improve model quality. Across a session, each change the agent kept to the training script made the next model better at predicting its dataset.
  4. It takes a lot of data and time to build big models. These small models trained for ten minutes on at most a few billion characters. The large models people use train on trillions of tokens for weeks or months.