Stemming vs lemmatisation, and why Croatian needs the second

Anything that counts words, from old topic models to the labels c-TF-IDF puts on modern ones, has the same problem: one word comes in many forms. “Study”, “studies” and “studied” are one idea, but to a counter they’re three unrelated strings, and each gets a third of the count. So text is usually normalised first, every form reduced to one. There are two ways to do it.

Stemming: chop off the ending

A stemmer follows simple rules to cut suffixes off, without understanding the word. It’s fast and needs nothing but a list of endings, and it often gets things wrong:

• running, runs → run: fine.
• studies, studying → studi: not a real word, which makes labels hard to read.
• boring → bore: a different word with a different meaning.
• better → better: it can’t know this is a form of “good”.

Lemmatisation: look up the dictionary form

A lemmatiser finds each word’s lemma, the form you’d look up in a dictionary, using a vocabulary and grammar rules, and often the word’s role in the sentence (noun or verb?). studies → study, better → good, was, were, is → be, and boring used as an adjective stays boring. It’s slower, and it needs a model or dictionary for each language, but what comes out is real words with the right meaning.

Why it matters so much for Croatian

English has only a few forms per word. Croatian has seven cases in two numbers for every noun, and verbs that change by person, tense and gender, and the stem itself often changes along the way. A stemmer that only chops endings can’t follow that:

For a regular noun like kuća (house), chopping endings works: all six forms become kuć. Not a word, but at least one string. For pisati (to write) the stem changes from pis- to piš-, so the stemmer splits one verb into two, and a topic about writing gets two half-strength labels. A lemmatiser maps every form to pisati. Tools exist for exactly this: CLASSLA, for example, is a pipeline for Croatian and other South Slavic languages that does lemmatisation along with tagging each word’s role.

What to use where

• For counting and labelling (LDA, c-TF-IDF labels, keyword lists): lemmatise. A review of 189 topic-modelling studies found stemming routinely made topics harder to read, and its advice was simply: don’t stem.
• For embeddings: don’t normalise at all. Embedding models were trained on real text and understand word forms themselves; feeding them lemmas only removes information.

So in a pipeline like BERTrend, the articles are embedded as they are, and only the copy of the text used for topic labels is lemmatised.