A computer cannot compare meanings, but it can compare lists of numbers. An embedding turns a word, a sentence or a picture into such a list: a point in space, placed so that things with similar meaning land close together. "Find related things" then becomes geometry. Measure angles, find the nearest points, and do it fast enough to search millions. Every vector on this page is a toy, built by hand or computed right here so you can check it. Real embeddings come from a trained model and have hundreds or thousands of dimensions.
Describe every word by answering the same 9 questions with a number. Is it a person? An animal? Royal? Female? Young? Plural? Big? Food? Sweet? Those 9 answers are the word's coordinates, so each word is a point in 9-dimensional space, and words that answer alike sit close together. Click two words to compare them, or build an analogy underneath. The picture is a flat shadow of the 9-D space. Every number on the right is computed in all 9 dimensions.
How to read this: each dot is a word. Words that answer the 9 questions alike sit close together. This is a flat shadow of 9-D space, so some words that look close are really far apart.
Try this: click "lion − man + woman" and watch the amber arrow land near the gold-ringed answer.
How to read this: one row per question. Bars grow right of the small center tick for yes and left for no. The last column multiplies A by B, and the gold total is the dot product.
The dot product adds up, axis by axis, how much two words agree (the last column of the table). Dividing by both lengths leaves only direction: cosine 1 means the same direction, 0 means nothing in common, −1 means opposite.
The best-overall view is a principal component projection. Subtract the average word, form the 9×9 covariance matrix C = XᵀX / n, and keep its two eigenvectors with the largest eigenvalues (found here by power iteration). Those two directions keep — of the total variance, so some words that look close in the picture are far apart in 9-D.
Every view projects onto two directions u and v at right angles: two eigenvectors, two named axes, or the analogy plane (one direction along a − b, the other along the part of c − b square to it). The note under the chart gives each view's share. Because projection is linear, a parallelogram in 9-D stays a parallelogram, though it can flatten to a line if the view hides one of its steps.
Here king − man + woman lands exactly on queen because the vectors were set up that way. Queen differs from king only on the female axis, just as woman differs from man. The search skips the three input words, which is the usual convention.
A real model learns its numbers from huge amounts of text (C05 · How machines learn). Its axes have no names, a single direction rarely means one clean thing, and analogies come out only approximately. The word2vec models from 2013 made this king and queen example famous.
import numpy as np
# 9 axes: person animal royal female young plural big food sweet
W = {"king": np.array([1, 0, 1, -.5, 0, 0, 0, 0, 0]),
"queen": np.array([1, 0, 1, .5, 0, 0, 0, 0, 0]),
"man": np.array([1, 0, 0, -.5, 0, 0, 0, 0, 0]),
"woman": np.array([1, 0, 0, .5, 0, 0, 0, 0, 0]),
"boy": np.array([1, 0, 0, -.5, 1, 0, 0, 0, 0]),
"cat": np.array([0, 1, 0, 0, 0, 0, .2, 0, 0])}
def cos(a, b):
return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))
def nearest(v, skip):
return max((w for w in W if w not in skip), key=lambda w: cos(v, W[w]))
v = W["king"] - W["man"] + W["woman"]
print(nearest(v, {"king", "man", "woman"})) # queen
X = np.array(list(W.values()))
Xc = X - X.mean(axis=0) # centre
vals, vecs = np.linalg.eigh(Xc.T @ Xc / len(X)) # covariance eigenvectors
xy = Xc @ vecs[:, ::-1][:, :2] # 2-D coordinates to plotSemantic search turns your question into a vector with the same recipe used for the documents, then returns the documents whose vectors point most nearly the same way. Try "kitten". No sentence contains that word, yet the meaning vectors still find the animal sentences. Switch to keyword vectors (TF-IDF) and the same question finds nothing at all.
How to read this: documents are ranked by how closely their meaning points the same way as your query. The bar shows the score. Green chips are query words the model has a vector for.
Try this: search "python snake", then "python code", and watch the top results swap.
Without normalizing, Euclidean distance also cares about length, and a long sentence makes a long vector. Normalize every vector to length 1 and distance becomes a decreasing function of cosine, so both metrics give exactly the same ranking. Many systems store unit vectors and use the plain dot product for this reason.
Here N = 18 documents and df is the number of documents containing the word. A word that appears everywhere gets weight 0.
Retrieval-augmented generation (RAG) is this search followed by one more step. The top k documents are pasted into the prompt, and a language model writes its answer from them (C07 · How a language model writes). The model only sees what retrieval found, so a bad search means a bad answer.
import numpy as np, math
# toy topic vectors [animal, food, travel, tech]; the page uses 8 topics
CONCEPT = {"cat": [1, 0, 0, 0], "kitten": [1, 0, 0, 0], "mouse": [.7, 0, 0, .5],
"bread": [0, 1, 0, 0], "soup": [0, 1, 0, 0], "train": [0, 0, 1, 0],
"laptop": [0, 0, 0, 1], "keyboard": [0, 0, 0, .8]}
docs = ["the cat chased a mouse", "bread and soup", "the train leaves at noon",
"my laptop keyboard and mouse"]
def embed(text): # sum of the known words' vectors
return sum((np.array(CONCEPT[w], float) for w in text.split() if w in CONCEPT),
np.zeros(4))
def tfidf(text, docs): # count x ln(N / df), no stop-word list here
vocab = sorted({w for d in docs for w in d.split()})
df = {w: sum(w in d.split() for d in docs) for w in vocab}
return np.array([text.split().count(w) * math.log(len(docs) / df[w]) for w in vocab])
def search(q, D, k, metric="cos", normalize=False):
if normalize:
q = q / np.linalg.norm(q)
D = D / np.linalg.norm(D, axis=1, keepdims=True)
if metric == "cos":
s = D @ q / (np.linalg.norm(D, axis=1) * np.linalg.norm(q))
return np.argsort(-s, kind="stable")[:k]
return np.argsort(np.linalg.norm(D - q, axis=1), kind="stable")[:k]
D = np.array([embed(d) for d in docs])
print([docs[i] for i in search(embed("kitten"), D, 2)]) # cat, laptop
T = np.array([tfidf(d, docs) for d in docs])
print(tfidf("kitten", docs).any()) # False: no shared wordBrute force compares the query with every stored vector. That is exact, and with millions or billions of vectors it is too slow to do for every search. A vector database builds an index ahead of time, so each search checks only a small part of the data and accepts that it may miss a few true neighbors. Here are 4,000 points in 2-D. Drag the gold query and watch how many distances each method computes and how many of the true nearest neighbors it finds.
How to read this: the gold cross is your query. Amber shows every point the search actually measured. IVF groups the points into clusters and searches only the n_probe clusters nearest the query. The graph walk hops from neighbor to neighbor and keeps a shortlist of the ef best points so far.
Try this: drag the gold query onto a cluster border with n_probe at 1 and look for red rings. Then raise n_probe to 3.
How to read this: up and to the left is best: more of the true k nearest neighbors found (that share is "recall@k") for less work. Each dot is one slider setting, and brute force is perfect but does all 4,000 comparisons.
IVF (inverted file) first runs k-means. It picks K points as centers, assigns every point to its nearest center, moves each center to the mean of its points, and repeats until no point changes cluster. Each round can only lower the total squared distance J = Σ |xi − ca(i)|². —
A search measures the query against the K centers, then scans only the n_probe closest lists. A true neighbor is missed when it sits just across a cluster border.
The graph walk links every point to its 6 nearest neighbors plus one random long link, both ways. The search starts at a fixed entry point and keeps stepping to neighbors closer to the query, holding a shortlist of the best ef points so far. HNSW (hierarchical navigable small world), an index many vector databases offer, adds sparser upper layers so the walk starts close to the target. This demo uses one layer.
Two dimensions make this easy. Here are 1,000 random points spread evenly in a d-dimensional cube, with one random query, computed on this page:
As d grows, the nearest point is barely closer than the farthest one. Space-splitting tricks that work well in 2-D (grids, k-d trees) then end up checking almost everything. Real embeddings are not evenly spread noise. They cluster around structure, which is why approximate indexes still work, but they need more probes and longer shortlists than this 2-D picture suggests, and each system tunes that trade-off on its own data.
import numpy as np
def kmeans(X, K, seed=0):
rng = np.random.default_rng(seed)
C = X[rng.choice(len(X), K, replace=False)] # K distinct points
a = None
while True:
new = ((X[:, None] - C[None]) ** 2).sum(-1).argmin(1)
if a is not None and (new == a).all():
return C, a # nothing moved
a = new
C = np.array([X[a == j].mean(0) if (a == j).any() else C[j]
for j in range(K)])
def ivf_search(q, X, C, a, k, n_probe):
near = np.argsort(((C - q) ** 2).sum(1))[:n_probe] # K distances
ids = np.flatnonzero(np.isin(a, near)) # probed lists
d = ((X[ids] - q) ** 2).sum(1) # len(ids) more
return ids[np.argsort(d)[:k]], len(C) + len(ids)
X = np.random.default_rng(1).random((4000, 2))
C, a = kmeans(X, 64)
found, cost = ivf_search(np.array([.5, .5]), X, C, a, k=10, n_probe=2)