Lemmatization and stemming · Normalization · Ayla's learning materials

Key pointsL'essentiel

Lemmatization groups the forms of one word under its dictionary form, where stemming only cuts endings by rule.
A stemmer works on letters alone, so each of its rules is a balance: applied too widely, it merges different words; applied too narrowly, it splits the forms of one word.
The lemma of a form often depends on its grammatical class, so lemmatization and part-of-speech tagging are usually done one after the other or together.

Lemmas and stems

Two normalizations group the inflected forms of a word: lemmatization and stemming.

linguisticslinguistique DefinitionDéfinition · lemmalemme · fr: lemme· en : lemma

The base or dictionary form under which the inflected forms of a word are grouped: go, goes, went and gone share the lemma go.Forme de base, ou forme de dictionnaire, sous laquelle sont regroupées les formes fléchies d'un mot : va, allons, allait et allé partagent le lemme aller. (Jurafsky & Martin, 2024, §2.6.1, p. 24)

NotesRemarques Corpus tools use lemmatization to count or search all forms of a word at once; a frequency list of lemmas answers a different question than a frequency list of word forms.Les outils de corpus utilisent la lemmatisation pour compter ou chercher toutes les formes d'un mot à la fois ; une liste de fréquences de lemmes répond à une autre question qu'une liste de fréquences de formes.

Other attested French terms: forme canoniqueAutres termes français attestés : forme canonique

NLPTAL DefinitionDéfinition · stemmingracinisation · fr: racinisation· en : stemming

A rough form of normalization that cuts the endings off words by rule, so that the forms of one word usually share a stem, which need not be a word of the language.Forme grossière de normalisation qui coupe les terminaisons des mots selon des règles, de sorte que les formes d'un mot partagent le plus souvent une racine, qui n'est pas forcément un mot de la langue. (Jurafsky & Martin, 2024, §2.6.1, p. 24; Manning et al., 2008, §2.2.4, pp. 32–33)

NotesRemarques A lemmatizer returns a dictionary form, a stemmer what is left once the ending is cut: chercher and cherches become cherch. Stemming needs no dictionary and errs in both directions, grouping forms that should stay apart and separating forms that belong together.Un lemmatiseur renvoie une forme de dictionnaire, un raciniseur ce qui reste une fois la terminaison coupée : chercher et cherches deviennent cherch. La racinisation n'a pas besoin de dictionnaire et se trompe dans les deux sens, en regroupant des formes à garder distinctes et en séparant des formes qui vont ensemble.

Other attested French terms: indexation par radicauxAutres termes français attestés : indexation par radicaux

	Lemmatization	Stemming
Result	A dictionary form: va, allons, allait become aller	What is left once the ending is cut, often not a word: chercher, cherches become cherch
What it needs	A dictionary or a model of the language, often the word’s tag	A list of endings to cut, for one language
Errors	Mostly on forms with two possible lemmas	Many, in both directions (Jurafsky & Martin, 2024)

Lemmatization and stemming compared.

Figure 3figure 3 shows both on nine French forms, with the stems the Snowball stemmer for French produces.The stems are the output of the Snowball stemmer for French (snowballstem.org), run on 6 October 2026.

Nine French forms in three groups, each with its lemma and its stem. livre and livres have the lemma livre and livrer the lemma livrer, but all three get the stem livr, marked as two words merged. chercher, cherches and cherchent all have the lemma chercher; chercher and cherches get the stem cherch, but cherchent stays cherchent, marked as one word split. va, allons and allait all have the lemma aller, but their stems are va, allon and allait, marked as one word split three ways. — Figure 3. Lemmas and stems for the same forms. The lemmatizer groups each word's forms; the stemmer merges two words and splits two others.

Stemming errors

A stemmer works on letters alone. It has no dictionary and no information about grammatical class, so nothing in its rules can tell that livre and livrer are two words, or that cherchent belongs with chercher. Writing its rules is a balance between two errors (Jurafsky & Martin, 2024):

Over-stemming: a rule applied too widely merges words that should stay apart, as livre and livrer under livr.
Under-stemming: a rule applied too narrowly, or left out, splits forms that belong together, as cherchent and chercher.

Applying a rule more widely removes some under-stemming and adds over-stemming, so every rule is a judgement about which does more harm.

Two endings, two decisions

In French, -ent ends the third person plural of the present and -ant the present participle, so removing them would bring more of a verb’s forms together. Both endings also end many other words, and the Snowball stemmer treats the two differently.

Ending	Rule	What it gains	What it costs
-ent	Never removed on its own	accident, moment, parent, souvent stay whole	cherchent stays apart from chercher: under-stemming
-ant	Removed	cherchant joins chercher under cherch	restaurant joins restaurer under restaur, important joins importer under import: over-stemming

Two verb endings in the Snowball stemmer for French.

The stemmer’s documentation explains the first decision: removing -ent or -ons would cause “unacceptable overstemming”, as in accident and garçons.Snowball, French stemming algorithm (snowballstem.org), read 6 October 2026; the stems in the table were produced with it the same day. That is also why allons only becomes allon: a later rule deletes the final s, and the -on stays.

Lemmatization and tagging

A form can belong to two lemmas: livre is its own lemma as a noun (un livre) and a form of livrer as a verb (il livre), as saw is a form of see as a verb and its own lemma as a noun (Manning et al., 2008). Figure 4figure 4 shows two such forms.

Four short French phrases with the tag and lemma of the form in bold. un livre: NOUN, lemma livre. il livre: VERB, lemma livrer. For these two, the lemma is chosen by the tag: NOUN gives livre, VERB gives livrer. mon fils: NOUN, lemma fils. des fils électriques: NOUN, lemma fil. For these two, the lemma is chosen by the rest of the sentence: both are NOUN, so the tag cannot choose. — Figure 4. The same written form with two lemmas. For livre the tag is enough to choose the lemma; for fils only the rest of the sentence can.

Tag and lemma are linked in three ways:

The tag settles livre, which is why many systems run a part-of-speech tagger first and the lemmatizer after it (Müller et al., 2015) (see also the Part-of-speech tagging (including an example of machine learning) chapter) (voir aussi le chapitre L'étiquetage morphosyntaxique (y compris un exemple d'apprentissage automatique)).
The lemma can help choose the tag, for instance in German, where a plural noun phrase does not show its gender and only the lemma does. Systems that do both tasks together improve on both (Müller et al., 2015; Kondratyuk et al., 2018).
The tag does not settle fils, a noun either way: the singular fils (son) or the plural of fil (thread, or wire in des fils électriques). Only the rest of the sentence tells them apart, and lemmatizers built with neural networks read that context directly, which helps them most on such ambiguous words (Bergmanis & Goldwater, 2018).

Written in English. The French version was automatically generated and I have not revised it. Rédigé en anglais. La version française a été générée automatiquement et je ne l'ai pas révisée. Updated 6 October 2026

Key pointsL'essentiel

La lemmatisation regroupe les formes d’un mot sous sa forme de dictionnaire, alors que la racinisation ne fait que couper des terminaisons selon des règles.
Un raciniseur ne travaille que sur les lettres, si bien que chacune de ses règles est un équilibre : appliquée trop largement, elle fusionne des mots différents ; appliquée trop étroitement, elle sépare les formes d’un même mot.
Le lemme d’une forme dépend souvent de sa classe grammaticale : lemmatisation et étiquetage morphosyntaxique se font donc d’ordinaire l’un après l’autre, ou ensemble.

Lemmes et racines

Deux normalisations regroupent les formes fléchies d’un mot : la lemmatisation et la racinisation.