Topic models before embeddings: LDA and its time-aware cousins

On the MEDIANA project I’m looking at how to spot new topics in the news as they appear. Before the modern methods built on embeddings (like BERTrend), there’s the classic family they grew out of: topic models. They’re still everywhere, and understanding why they fall short tells you what the newer methods had to fix.

LDA: words that keep turning up together

Latent Dirichlet Allocation (LDA, 2003) treats every document as a bag of words: order and meaning are ignored, only which words appear. Its assumption is that each document is a mix of a few topics, and each topic is a set of words that tend to be used together. You tell it how many topics you want (K), and it works out which words belong to which.

It gets there by starting from a random guess and then fixing it one word at a time. For each word it temporarily removes its label and asks two questions:

1. Which topic is this document mostly about right now?
2. Which topic already uses this word elsewhere?

The word moves to the topic that both answers favour. Repeat for every word, many times, and words that share documents drift into the same topic. Here are four tiny documents and K = 2:

The animation is simplified in one way: real LDA (with Gibbs sampling) samples the new topic with those odds instead of always taking the more likely one, which helps it escape bad starting points on real data. The two questions are the real ones, though: the odds are proportional to how much the document uses the topic times how much the topic uses the word.

LDA has well-known weaknesses. You have to choose K yourself. It sees words, not meaning: “car” and “automobile” are as different as “car” and “banana”, the problem embeddings solve. And it needs plenty of words per document to see what co-occurs, so it works poorly on short texts like tweets or headlines. That last one matters: a review of 189 social media studies found that 80% still used LDA, on exactly the kind of short text it handles worst, largely because everyone else did.

Dynamic topic models: following topics through time

LDA treats all documents as if written at once. Dynamic topic models (DTM, 2006) cut the data into time slices, say years, and keep one version of every topic per slice. They add a third question to LDA’s two:

3. How close is this year’s version of the topic to the years around it? Small changes between neighbouring years are cheap; big jumps are expensive.

That pull between neighbouring years needs balance. Too strong, and the topic can’t follow real change; too weak, and “the same topic” in two years is two unrelated lists of words. Get it right and you can watch topics rise and fall, and follow individual words within a topic.

Continuous-time DTM (cDTM, 2008) drops the fixed slices: the allowed change grows with the time that actually passed between documents. An hour apart, little can change; a year apart, a lot can. Quiet periods cost nothing, so you can choose the time resolution that suits the data rather than the one your computer can handle. That idea, scaling change by elapsed time, fits news well, where some days bring hundreds of articles and some bring none.

The catch: a fixed number of topics

All of these models have exactly K topics, from the first time slice to the last. A topic can grow, shrink and change its words, but a genuinely new topic has nowhere to go: its documents get squeezed into whichever existing topics fit least badly. For following known topics through time they’re a good foundation. For noticing something new as it emerges, which is what MEDIANA is after, they’re the wrong tool, and that’s where clustering-based methods come in: embeddings grouped into topics, and BERTrend on top of them.

Lessons from 189 studies

The same review is a useful checklist for any topic modelling, old or new:

• Remove the words you searched for. Collect tweets with #gillette, keep “gillette” in, and it dominates every topic. Only about 19% of the studies on tweets removed them.
• Don’t stem. Stemming chops off endings by rule (“studies” → “studi”, “boring” → “bore”), making non-words and merging different meanings. Lemmatising (its own note) looks up the dictionary form (“studies” → “study”, “better” → “good”). That matters a lot for Croatian, where one word can have a dozen forms.
• Don’t trust a single score. Coherence and perplexity are easy to compute but don’t tell you if topics are useful. Read example documents, not just the top ten words.
• Report your choices. 62% of the studies didn’t even say how they split text into words.