Spotting emerging topics with BERTrend

The question at the heart of MEDIANA: across a constant stream of news, which topics are new and growing right now? Not the big stories everyone already covers, and not one-off articles that go nowhere, but the early signs of something about to matter. One of the most promising approaches I’ve read is BERTrend (2024). It builds on BERTopic, which finds topics in one batch of documents, and adds what’s needed to follow them through time and judge whether they’re rising.

Step by step

Steps 2–4 have a note of their own, with pictures of HDBSCAN finding groups and c-TF-IDF scoring words.

1. Embed every document (a news article, or a chunk of one) with a language model, as in RAG: similar meaning, nearby points (how embeddings work).
2. Shrink the embeddings to about 5 dimensions with UMAP. Clustering works badly in hundreds of dimensions, where every point is about equally far from every other. UMAP keeps neighbours as neighbours while throwing away most dimensions.
3. Cluster with HDBSCAN, which finds dense groups of points without being told how many to look for, and leaves stray points as outliers. Each cluster is a topic. It’s tuned to allow many small topics, and that’s a balance: too coarse and a small new story gets swallowed by a big neighbour; too fine and you drown in hyper-specific fragments.
4. Label each cluster with c-TF-IDF: the words frequent in this cluster but rare in the others. A cluster of articles about sheep, a virus serotype and ruminants gets the label “farmers, serotype, ruminant, infected”.

Then BERTrend takes over, for every new time slice:

5. Merge with the past. Each new topic is compared with the existing ones. If its similarity to one of them is 0.7 or more, it’s the same topic continuing and gets merged in; below that, it’s new. A topic keeps the embedding it was born with, so a long-running topic can’t slowly drift into meaning something else.
6. Update popularity. A topic’s popularity goes up by the number of documents it gets in each slice, and when it gets none, it decays exponentially. Topics nobody writes about fade by themselves.
7. Classify each topic by its popularity compared with all the others, and by its trend:

Popularity	Trend	Label
below the 10th percentile	any	noise
10th–50th percentile	rising	weak signal: an emerging topic
10th–50th percentile	falling	noise
above the 50th percentile	any	strong signal: already big

The weak signals are the point of the whole pipeline: small enough that nobody’s noticed, growing fast enough to matter. Using percentiles rather than fixed numbers means the thresholds adapt to how busy the news is.

Telling it what to look for

BERTrend also has an expert-guided zero-shot mode. Instead of waiting for HDBSCAN to form a cluster, an analyst writes a short description of what they care about, say “outbreaks of animal disease in Europe”. The description is embedded too, and each document is compared with it directly; anything with similarity above a lower threshold (0.4–0.6, since a description is vaguer than an article) counts towards that topic, and the same popularity and trend rules apply. It catches a topic earlier, before there are enough documents for a cluster of its own. The catch: you have to know roughly what you’re looking for. It’s “zero-shot” because the expert gives a description, not examples.

Letting a language model explain it

The output so far is clusters of keywords and popularity curves, which is not what a journalist wants to read. So BERTrend hands each topic’s keywords and some example articles to an LLM, which writes a title, the key developments and a “what’s new since last time” section for every time slice; then, from those summaries, a deeper analysis: short- and long-term impact, optimistic and pessimistic scenarios, related trends, and what could speed it up or slow it down. In the paper’s example, a bluetongue virus outbreak becomes a readable timeline, from France setting up restricted zones for livestock to a new serotype spreading across Europe.

What I’m keeping an eye on

• The settings matter a lot: the cluster sizes, the 0.7 merge threshold, the percentiles. They have to be tuned on our own data, not copied from the paper.
• Croatian. The embedding model has to understand it well, and the labels need lemmatised words, or one topic gets ten spellings of the same noun.
• How do you evaluate it? There’s no list of “correct” emerging topics to compare against. One answer is to replay past news and check whether the pipeline flagged stories before they got big, and to use an LLM as a judge for the rest.