From embeddings to topics: UMAP, HDBSCAN and c-TF-IDF

Old topic models count which words appear together. The modern way, used by BERTopic and so by BERTrend, starts from meaning instead: turn every document into an embedding, find the groups of documents that sit close together, and name each group by its words. It takes three tools, one per step.

1. UMAP: fewer dimensions, same neighbours

Embeddings have hundreds of dimensions, and that’s a problem for finding groups. In so many dimensions, distances between points become very similar to each other, so “dense” and “sparse” stop meaning much. UMAP (Uniform Manifold Approximation and Projection) squeezes the points down to a few dimensions (BERTopic uses 5 for clustering, and 2 only for drawing pictures) while trying to keep each point’s nearest neighbours as its nearest neighbours.

Roughly how: it builds a graph connecting every point to its k nearest neighbours in the original space, then arranges points in the small space so that the same graph still holds. Two settings matter most. n_neighbors sets how many neighbours each point listens to: small values keep fine local detail, large ones the overall shape. min_dist sets how tightly points may pack. The price is that only neighbourhoods are kept: how far apart two groups end up says little about how different they really are.

2. HDBSCAN: dense groups, and the courage to say “neither”

Now find the groups. A method like k-means needs to be told how many groups there are, and puts every point in one. Neither fits news: nobody knows how many topics today’s articles hold, and plenty of articles belong to no topic at all.

HDBSCAN looks for dense regions instead. In short: it measures how crowded each point’s neighbourhood is, links points through crowded areas, and keeps the groups that stay together over a wide range of density levels. Points in thin, scattered areas are left out as outliers (topic −1 in BERTopic) rather than forced into the nearest topic.

The key setting is min_cluster_size: the smallest group that counts as a topic. Set it low and you get many small, specific topics (and some noise); set it high and you get a few broad ones, where a small new story may be swallowed by a big neighbour. For emerging topics, where the interesting stories start small, that choice matters a lot.

3. c-TF-IDF: naming each group

A cluster is just a set of points; people need words. BERTopic treats all the documents of one cluster as one long document and scores each word with a variant of TF-IDF, class-based TF-IDF:

score(word, topic) = how often the word appears in the topic × log(1 + average words per topic ÷ how often the word appears in all topics)

The first part rewards words the topic uses a lot. The second punishes words every topic uses. So “the” and “said” score near zero everywhere, while “serotype” scores high in exactly one topic.

The top few words become the topic’s label: “sheep, serotype, vaccine, farmers”. It’s a good start, but still just a list of words. That’s why BERTrend hands each topic’s words and example articles to an LLM to write a readable title and summary.

Things to watch

• Lemmatise before labelling. In Croatian one noun can have a dozen forms, and c-TF-IDF counts each separately. The embeddings can use the original text; the words for labels should be reduced to their dictionary forms first.
• Run-to-run differences. UMAP has randomness in it: two runs give slightly different maps, and so slightly different topics. Fix the random seed, and check that the topics that matter survive a few runs.
• Outliers aren’t failures. A large share of documents in topic −1 can just mean the news that day was scattered. It can be reduced afterwards by assigning outliers to their closest topic, if every document needs a home.