Tiny Training

Make subtitle h6 (first of hierarchy after text), make blockquote larger. Python… executed, then stdout. Or, executed in my editor, then written to .blocks/{}, then previewd in nvim with transclusion, then rendered as stdout and parsed as JSON to render in a client-side view.

Also side-bar needs a full-height border and toggle. freez at top for mobile header.

TinyStories is a 10M-parameter model by Microsoft. Here’s the abstract:

In this work, we introduce TinyStories, a synthetic dataset of short stories that only contain words that a typical 3 to 4-year-olds usually understand, generated by GPT-3.5 and GPT-4. We … train and evaluate LMs that are … below 10 million total parameters … [which] produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate reasoning capabilities.

We also introduce a new paradigm for the evaluation of language models: We suggest a framework which uses GPT-4 to grade the content generated by these models as if those were stories written by students and graded by a (human) teacher.

Data first

We’ll fetch the data[^Use a HF prefix chip to link] from HuggingFace’s datasets, then take a quick peek.

!curl -L https://huggingface.co/datasets/roneneldan/TinyStories/resolve/main/TinyStories-valid.txt -o data/tiny-stories.txt
!head !$

Looks like stories are split by <|endoftext|>

contents = open("data/tiny-stories.txt").read()
stories = [story.trim() for story in contents.split("<|endoftext|>")]

Introducing

@model(“GPT-4”)

[@model(GPT-4)