Embeddings: meaning as a point in space

Almost everything in my other notes rests on one idea: turn a piece of text into a list of numbers, so that texts with similar meanings get similar numbers. That list is an embedding. It’s how RAG finds the right paragraphs, how BERTrend groups news articles into topics, and how a computer can tell that “registering an obrt” and “opening a sole proprietorship” are about the same thing while sharing no words at all.

A point in a space with many dimensions

An embedding model reads a text and outputs a vector: a list of a few hundred to a few thousand numbers. Emma’s model, OpenAI’s, gives 1,536. Think of each list as the coordinates of a point. During training, the model sees huge numbers of texts that belong together (a question and its answer, a title and its article, two translations of the same sentence) and is pushed to place them close, and unrelated texts far apart. What comes out is a space where direction means meaning.

Measuring “close”: cosine similarity

The usual measure is cosine similarity, the cosine of the angle between two vectors:

• 1: pointing the same way, the same meaning.
• around 0: at right angles, unrelated.
• −1: opposite. Real text embeddings rarely get there; “unrelated” is the usual floor.

It only looks at the angle, not the length, so a long article and a short headline about the same thing still match. Many systems scale every vector to length 1 first, and then cosine similarity, the dot product and plain distance all rank neighbours the same way. That’s why a vector index like FAISS can simply look for the nearest points.

Words, then sentences

The first well-known embeddings were for single words (word2vec in 2013, GloVe in 2014): every word got one fixed vector, famous for arithmetic like king − man + woman ≈ queen. Their weakness is that “bank” gets the same vector whether it’s a river bank or a savings bank. Modern models are transformers that read the whole sentence first, so each word’s representation depends on its context, and they can produce one embedding for a whole sentence, paragraph or article. That’s what RAG and topic models use today.

Multilingual models go a step further: they’re trained on translations, so “Kako otvoriti obrt?” and “How do I open a sole proprietorship?” land almost on top of each other. That matters for MEDIANA, where the news is in Croatian but many of the best models were trained mostly on English.

Seeing a space you can’t see

Nobody can picture 1,536 dimensions. To look at embeddings, tools like UMAP squash them down to 2 (or, for clustering, about 5) while keeping neighbours as neighbours. The maps are useful for spotting groups, but distances across the map are only roughly right: two clusters that look far apart on the page may not be, and the empty space between them means nothing.

What embeddings are used for

• Search by meaning: embed the question, find the nearest chunks. That’s RAG.
• Grouping: cluster the points and each group is a topic. That’s BERTopic and BERTrend, and what replaced counting words in older topic models.
• Matching against a description: embed a sentence like “outbreaks of animal disease” and compare everything with it, no examples needed (BERTrend’s zero-shot mode).
• Finding duplicates: the same story from ten outlets lands in nearly the same spot.

Where it falls short

• Similar isn’t the same as answering. “Who founded the firm?” isn’t phrased like the About page that answers it, so it lands far away. It’s a real failure from Emma, shown in the RAG note.
• Small words with big effects. Negation can barely move a sentence: “the deal was approved” and “the deal was not approved” can end up very close.
• Outside its comfort zone, like a language or a field it saw little of in training, the space gets fuzzy, and everything looks a bit similar to everything else.

Inside the language models themselves, words become vectors in the same way, plus one more ingredient: where each word sits in the sentence. That’s the subject of positional encoding.