What does it mean that intelligence is compression?

The best model of something is the shortest one that still reproduces it. Understanding is grasping the most with the fewest words. It is a real idea from information theory, behind Kolmogorov complexity, minimum description length, and even a prize for compressing Wikipedia.

What is the five-words exercise?

Imagine your whole life with only five words you can say, one at a time, alone. The words you reach for are not random, they are whatever is most loaded right now, so the set becomes a compressed, honest summary of where your head is.

What do tokens have to do with this?

Tokens are the small pieces a language model breaks the world into before it rebuilds it. They are LEGO bricks for meaning. You pick the bricks, and the bricks decide what you can build.

Tokens are LEGO bricks: Your Life in Five Words

Intelligence is compression

The less word you use to explain something complicated, the smarter you are.

(Based on my reading of LLM equations)

We all know that guy who makes every explanation sound easy…

The reverse is also true

Understanding is how much you can capture without needing many words.

Then my brain went somewhere I did not expect.

Five words

Imagine you live a life with only five words in your dictionary. You can say them one at a time, alone.

What would they be?

Sit with it before you read mine.

It is harder, and stranger, than it sounds.

Mine came out as this

I was eating ice cream on a Sunday. No prompt, no list. Five words just arrived, in this order:

grateful
go
see
eat
coding

Five words make about 325 little sentences once you allow every length and every order.

from itertools import permutations

words = ["grateful", "go", "see", "eat", "coding"]

# every ordering, every length, no punctuation
for n in range(1, len(words) + 1):
    for p in permutations(words, n):
        print(" ".join(p))
# prints 325 of them

The first ten it prints:

grateful
go
see
eat
coding
grateful go
grateful see
grateful eat
grateful coding
go grateful

Funny, the less word the more meaning…

The fun part

Now let’s give AI the words and guess the sentence:

The tokens	What it reads as
grateful, coding	thankful for the work itself
go, see, eat	move, look, take it in
see coding	I watch, then I build
grateful, go	thankful, and still moving
eat, coding	fuel, then make
go, see, coding	explore, notice, build it

Turn all these sentences into one

Now take all 325 permutation sentences and compress all the sentence into one:

A grateful person who keeps moving, pays attention to the world and turns what he sees into things he builds.

That’s scary but yeah, pretty much what I do daily…

What would be yours?

Tokens are just LEGO bricks

That is roughly the definition of a token.

The world broken down into a small piece by a math function.

So that another math function can rebuild the world.

Compression.