Predicting the next word · LLMs as next-word predictors

Autocomplete

Type Good into a message on your phone. Your keyboard will suggest the next words it predicts to be most likely: morning, night, luck. Add one letter, Good m, and the list narrows to morning. Nothing in the keyboard understands what you are writing. It is answering a single question: given the words already typed, which word is likely to come next?

language modelsmodèles de langue DefinitionDéfinition · language model (LM)modèle de langue · fr: modèle de langue· en : language model

A machine-learning model that estimates how likely different sequences of language are.Modèle d'apprentissage automatique qui estime la probabilité de différentes séquences de langue. (Jurafsky & Martin, 2026, p. 68)

NotesRemarques Given a sequence of tokens as context, it assigns probabilities to the tokens that could come next. These probabilities can be used to predict or generate text and to compare the likelihood of complete sequences.À partir d'une séquence de jetons donnée en contexte, il attribue des probabilités aux jetons qui pourraient suivre. Ces probabilités servent à prédire ou à générer du texte et à comparer la vraisemblance de séquences complètes.

Other attested French terms: modèle de langageAutres termes français attestés : modèle de langage

language modelsmodèles de langue DefinitionDéfinition · next-token predictionprédiction du jeton suivant · fr: prédiction du jeton suivant· en : next-token prediction

The task of estimating the probabilities of the tokens that could follow a given sequence of tokens.Tâche consistant à estimer les probabilités des jetons qui pourraient suivre une séquence de jetons donnée. (Jurafsky & Martin, 2026, pp. 5–9)

NotesRemarques It is a common training objective for autoregressive language models. During text generation, the model repeatedly produces this probability distribution, a decoding method selects a token, and the selected token is added to the context for the next prediction.C'est un objectif d'entraînement courant des modèles de langue autorégressifs. Pendant la génération de texte, le modèle produit cette distribution de probabilité de façon répétée, une méthode de décodage choisit un jeton, et le jeton choisi s'ajoute au contexte pour la prédiction suivante.

Other attested English terms: next-word predictionAutres termes anglais attestés : next-word prediction

Your keyboard answers from something small. The older way is a frequency table: the keyboard has counted which words followed Good, in the text it was built from and in your own past messages, and suggests the most frequent one. Current keyboards run a small neural network instead, with a vocabulary of around ten thousand words and about a megabyte of storage on the phone (Hard et al., 2018). Either way it looks one or two words behind the cursor.

Click Figure 1figure 1 to test the quality of the autocomplete on my phone.

Figure 1. My own phone, 21 November 2023. One tap of the middle suggestion per stage.

The same question with more behind it

Compare the results from the autocomplete in Figure 1figure 1 with what we have come to expect from large language models (LLMs) like ChatGPT, Claude, Gemini, etc.

language modelsmodèles de langue DefinitionDéfinition · large language model (LLM)grand modèle de langue (GML) · fr: grand modèle de langue· en : large language model

A neural-network language model, usually based on the transformer architecture, that is trained on very large amounts of text and contains many learned parameters.Modèle de langue à réseau de neurones, habituellement fondé sur l'architecture des transformeurs, entraîné sur de très grandes quantités de texte et comptant de très nombreux paramètres appris. (Jurafsky & Martin, 2026, pp. 5–7)

NotesRemarques Most current generative LLMs process text as tokens and generate text one token at a time by repeatedly assigning probabilities to possible next tokens. When incorporated into an application, an LLM can support tasks such as answering questions, summarizing, translating, and conducting conversations.La plupart des GML génératifs actuels traitent le texte sous forme de jetons et le génèrent un jeton à la fois, en attribuant de façon répétée des probabilités aux jetons suivants possibles. Intégré à une application, un GML peut servir à répondre à des questions, à résumer, à traduire ou à mener des conversations.

To understand the difference in quality, consider these important differences:

LLMs are far bigger. The keyboard’s model is around a million adjustable numbers, small enough to sit on the phone in your hand. LLMs have billions of them, and they run on hardware you never see.
LLMs learned from far more text, written by other people. Your keyboard learned from a modest word list and from your own past messages. LLMs are trained on a very large and varied collection of text. Their vocabulary, their phrasing and their factual associations all come from there.
LLMs read far more of what you already typed. Where the keyboard uses the last word or two, they take thousands of tokens of preceding text into account, so a pronoun twenty sentences back can still affect what they predict now.
LLMs represent each word far more richly, and not the same way twice. Both turn words into word embeddings, vectors of numbers standing for a word (see also the Word embeddings chapter, in the Language as numbers module) (voir aussi le chapitre Plongements lexicaux, dans le module Le langage en nombres). The keyboard’s are about ninety-six numbers long and fixed once, so bank is the same vector in every sentence you type. An LLM’s run to thousands of numbers and are recomputed for the sentence at hand by the attention mechanism, which decides which of the earlier words matter for the word being predicted now (see also the Attention & transformers chapter, in the Language as numbers module) (voir aussi le chapitre Attention et transformeurs, dans le module Le langage en nombres).

One objective to solve many tasks

While LLMs are trained just to predict the next most probable token, that single objective manages to create systems that are good at many tasks. This is quite different from the previous generation of NLP systems, where we had more specialized dedicated tools per task. Instead of one translation model, one summarizing model, one code completion model, we now have one giant LLM able to do it all.

Four requests on the left, translate this paragraph, summarize this report, write a function, and etc., each feeding into one tall box marked next-token prediction, which sends back a translation, a summary, a function, and etc. — Figure 2. Translation, summary and code come out of the same prediction, asked in different contexts.

Written in English. The French version was automatically generated and I have not revised it. Rédigé en anglais. La version française a été générée automatiquement et je ne l'ai pas révisée. Updated 9 September 2026

La saisie automatique

Tapez Good dans un message sur votre téléphone. Votre clavier vous proposera les mots qu’il juge les plus probables ensuite : morning, night, luck. Ajoutez une lettre, Good m, et la liste se réduit à morning. Rien dans le clavier ne comprend ce que vous écrivez. Il répond à une seule question : étant donné les mots déjà tapés, quel mot a le plus de chances de suivre ?