Meaning as a Vector

"Meaning" sounds abstract. Computers do math, not meaning. The trick: you map meaning onto numbers. That mapping lives in a vector space.

Two dimensions are enough for an intuition

Imagine a map with two axes: size on x, yellowness on y.

• A mouse is small and not yellow → bottom left.
• A giraffe is tall and slightly yellow → top right.
• A honeybee is small and very yellow → top left.

Try it yourself: type a name above and drag the ★ to where you'd put the word.

What you just did: assigned each animal a point in 2D space describing its meaning along those two axes. Animals with similar meaning sit close together. That's the kernel of every embedding.

More dimensions, more world

Two dimensions aren't enough. Lives in water? Mammal or not? Calm or dangerous? Diurnal or nocturnal? Each becomes its own axis - and suddenly you can really tell animals apart.

A modern embedding model assigns not 2 or 4 but hundreds or thousands of dimensions. Nobody tells the model what each dimension means; they emerge during training as whatever best separates concepts.

Embedding models: size vs. quality

Three models we use in the lab:

More dimensions usually means more nuance - but also more storage, more latency, bigger index.

Embedding models have context windows too

A non-obvious gotcha: embedding models also have a context limit, typically a few hundred to a few thousand tokens. The spread is bigger than you'd think - here are our three lab models:

A factor of 32 between the smallest and the largest window - and that matters in a moment. Anything past the limit gets silently truncated: no error, no warning, the rest just disappears. That's exactly why longer texts have to be cut up deliberately before embedding. That bridge is the next chapter.

From many axes down to two: PCA

We just said modern embedding models assign hundreds or thousands of dimensions. A screen has two. How do you get from 1024 axes down to two without lying outright?

The standard answer is PCA - Principal Component Analysis. PCA finds the two axes in the high-dimensional space along which the points spread the most - i.e. where the biggest differences in meaning sit - and projects every vector onto that plane. What's left is a 2D picture showing as much of the original structure as two axes can.

Important: this is a crutch for the eye. 1022 axes get thrown away. If the clouds in the lab look fuzzy, that's often less about the model itself than about this drastic reduction. The actual embedding is still 1024-dimensional and clean - only our view of it is flat.

Lab: embedding explorer

Lab 2 Embedding Explorer projects a curated set of terms from four domains - animals, vehicles, food, cities - into 2D via PCA. Each dot is exactly one term. You see how the four topics sort into their own clouds, even though the model was never told there are categories - it just knows the words.

Switch the model and watch the clusters drift: a bigger model often pulls the clusters apart more sharply, a smaller one lets them blur into each other. Try nomic-embed-text and watch what happens - for single words the points collapse almost on top of each other. That model is trained on longer passages; one token isn't enough for it to encode meaningful structure. Exactly the kind of trap that makes embeddings useless in real RAG setups.

Type your own word like "raptor" or "capital" and see where the point lands. Even that gives you an intuition for what the model means by "meaning".

What to watch for in particular: type a word that lives in none of the four clusters - say "flower" or "cloud" - and the model still hands you a "nearest neighbour", sometimes a surprising one, like "bread" for "flower". That's not a bug. For single words, cosine values tend to sit close together (typically 0.5-0.7), and the model picks "the least wrong option" rather than a real semantic match. Exactly the kind of result that produces bad answers in real RAG setups - and one reason we later embed whole sentences instead of single words.