Teach · From neural networks to large language models

Worked examples and detail elsewhere

This chapter explains the last two ways of building a language system: a network that learns its own features, and one model that is given its task in a prompt. It stays at the level of the idea. How text becomes numbers a network can read, how a large language model is trained and used, and what it can and cannot do each have a module of their own, starting with the next one (see also the Language as numbers module) (voir aussi le module Le langage en nombres); the classical NLP module shows what the earlier paradigms did on three tasks (see also the Classical NLP module) (voir aussi le module TAL classique).

The systems behind much of today’s language technology, including almost all machine translation systems and chatbots with which you will interact, are neural networks. The name suggests something brain-like, and the inspiration is real, but loose: there are some similarities between artificial neural networks and natural neural networks like our brain, but this does not go beyond the very basics.

Schematic of a small neural network: an input layer of circles on the left, two hidden layers in the middle, an output layer on the right, with lines connecting each unit to every unit in the next layer. — Figure 1. The standard picture of a neural network. Each circle is a unit, each line a weighted connection. Data enters on the left as numbers and comes out on the right as an answer.

machine learningapprentissage automatique DefinitionDéfinition · neural network (NN)réseau de neurones (RNA) · fr: réseau de neurones· en : neural network

A machine-learning model composed of interconnected layers of small computational units.Modèle d'apprentissage automatique composé de couches interconnectées de petites unités de calcul. (Jurafsky & Martin, 2026, p. 11)

NotesRemarques Each unit combines input values using adjustable numerical weights and passes the resulting value to other units. During training, the network adjusts its weights to improve its performance on a task. The name reflects the field's historical inspiration from biological neurons; modern neural networks are mathematical computing systems, not realistic models of how the human brain works.Chaque unité combine des valeurs d'entrée au moyen de poids numériques ajustables et transmet le résultat à d'autres unités. Pendant l'entraînement, le réseau ajuste ses poids pour améliorer sa performance sur une tâche. Le nom rappelle l'inspiration historique des neurones biologiques ; les réseaux de neurones modernes sont des systèmes de calcul mathématiques, et non des modèles réalistes du fonctionnement du cerveau humain.

Other attested French terms: réseau de neurones artificielsAutres termes français attestés : réseau de neurones artificiels

Each unit, called a neuron, is a small mathematical function. It receives numbers from the units connected to it, combines them, and passes a number on. Every connection has a strength, called a weight, and those weights, the network’s parameters, are what training adjusts: a stronger connection means one unit’s output matters more to the next.

machine learningapprentissage automatique DefinitionDéfinition · weightpoids · fr: poids· en : weight

A real number attached to one input of a computing unit, learned during training, that sets how much that input counts towards the unit's output.Nombre réel attaché à une entrée d'une unité de calcul, appris pendant l'entraînement, qui détermine l'importance de cette entrée dans la sortie de l'unité. (Jurafsky & Martin, 2026, pp. 11, 95)

NotesRemarques A weight can be positive or negative, so an input can count for or against a given answer. Weights and bias terms are the two kinds of number that make up a model's parameters, and the set of all of a network's weights specifies the network: that is what an open-weight model releases and a proprietary one withholds.Un poids peut être positif ou négatif, si bien qu'une entrée peut compter pour ou contre une réponse. Les poids et les biais sont les deux sortes de nombres qui composent les paramètres d'un modèle, et l'ensemble des poids d'un réseau spécifie ce réseau : c'est ce qu'un modèle à poids ouverts diffuse et qu'un modèle propriétaire retient.

Other attested French terms: pondérationAutres termes français attestés : pondération

The units are organized in layers. An input layer takes in the raw data as numbers; one or more hidden layers transform it, step by step; an output layer produces the answer, for example one value per possible category. When there are many hidden layers, the network is called deep, which is all that the “deep” in deep learning means (LeCun et al., 2015).

Two networks side by side, each with the same three input units and two output units. On the left, one hidden layer of four units, labelled shallow. On the right, four hidden layers of four units, labelled deep, with many more connections between them. — Figure 2. The same input and the same output, once with one hidden layer and once with four. Only the number of hidden layers changes.

Depth is what lets a network build complex patterns out of simple ones. Early layers pick up simple regularities in the raw input; later layers combine those into larger ones. In image recognition, the classic illustration, first edges, then shapes, then faces. For text the same principle applies, with one catch: a network’s input layer takes numbers, so using one for language requires a way to encode language as numbers (see also the Language as numbers module) (voir aussi le module Le langage en nombres).

Three columns of small tiles. On the left, tiles each containing one straight edge at a different angle. In the middle, tiles where two or three edges meet into a corner, an angle or a cross. On the right, tiles containing a closed circle and a closed triangle. Arrows run left to right between the columns. — Figure 3. A schematic, not real network output: simple motifs in the early layers, combinations of them in the middle, closed shapes deeper in.

Key pointsL'essentiel

Training strengthens or weakens connections based on feedback: every example pushes the parameters toward what would have been right, and a wrong prediction pushes hardest.
Networks learn their own features from raw data, which removed the ceiling of feature engineering, at the price of needing far more data.

A toy example

Suppose we want a network that tells diamonds, squares and circles apart.Shapes are a toy task, chosen for its simplicity and easy visualization. A real network chooses among far more classes, and its input is rarely anything as tidy as a drawn shape. The loop is the same either way, which is what lets the example be this small. That already fixes the shape of its output layer: one unit per category, one for square, one for circle, one for diamond. Their scores are turned into percentages that add up to 100, so the three answers compete, and raising one lowers the others.

A network answers in probabilities from the start. Untrained, they are essentially random. Given a picture of a diamond and asked what shape it is, the answer might be something like: square 40%, circle 35%, diamond 25%. Early in training they can be confidently wrong instead: square 80%, circle 10%, diamond 10%. The network is then told the right answer, and the predict-compare-adjust loop of supervised learning reappears in its neural form (see also the From rules to machine learning chapter) (voir aussi le chapitre Des règles à l'apprentissage automatique): the error flows backward through the network, and the connections that pushed toward square are weakened, the ones that pushed toward diamond strengthened.

Figure 4. The training loop. The network predicts, the prediction is compared with the label, and the connections are adjusted. Repeated over enough examples, the predictions sharpen.

Millions of small adjustments

Adjustment is not reserved for mistakes. A network nudges its connections on every example, including the ones it already gets right; the nudge simply gets smaller as the prediction moves closer to the label. One example changes little. Millions of examples, each nudging millions of parameters, add up to a system whose predictions are sharp. Nothing else in the network changes during training: the architecture, the layers and the connections stay as designed. Learning is entirely a matter of adjusting the strengths.

The difference from a feature-based model

The decisive difference from a feature-based model is what the network learns from. A feature-based model needed a human to decide that suffixes and capitalization matter. A neural network takes raw data (e.g., pixels of an image, or numerical representations of words), and finds its own regularities in it, including ones no human would have thought to specify. The features are learned, and feature engineering disappears. The human work does not disappear with it: choosing and encoding features used to be most of the effort of building a system, and the effort now goes into collecting the data, defining the task and designing the network.

Why it took until the 2010s

Nothing about the idea is new: the underlying design goes back to the perceptron of the 1950s (Rosenblatt, 1958), the first model that could learn its own weights from labelled examples (Goodfellow et al., 2016). In the 2010s, its two main requirements could finally be fulfilled. Learning features from raw data takes far more examples than learning weights for a few hand-picked features, and processing them takes enormous computation. Large collections of digital text and images supplied the first; graphics processors, built to render games, turned out to suit the calculations and supplied the second. With both in place, deep networks moved the state of the art in speech recognition, image recognition and object detection, and in fields as far from language as drug discovery (LeCun et al., 2015).

With a rule-based system you can always find out why an output came out as it did: some rule was triggered, and that rule can be read, disagreed with and changed. In a trained neural network there is nothing to read. Its behaviour is stored as millions of numerical connection weights, learnt from data, and no single one of them means anything on its own. The mathematics of every step is known exactly; what the steps collectively do is not stated anywhere in a form a person can check.

Statistical systems are a sort of middle ground: their rules are learnt from data rather than written by hand, but stay more or less readable (see also the Four paradigms page) (voir aussi la page Quatre paradigmes). How readable depends on the algorithm: a decision tree can be printed and followed, while the weights of a support vector machine with a non-linear kernel cannot be read that way.

Three panels reaching the same verdict, spam. On the left, rules written by hand: IF unknown sender AND links greater than 5 THEN spam, defined, read and changed by humans. In the middle, a statistical model listing the same kind of conditions with learnt weights beside them, readable to a degree that depends on the algorithm. On the right, a trained network showing a grid of weights such as 0.13, minus 0.42, 0.87, learnt from data and not interpretable. — Figure 5. The same verdict reached three ways: rules written by hand, rules learnt from data, and weights learnt from data.

This is what people mean when they call neural networks black boxes, and it is a property of the approach rather than a defect to be patched later. The power of the method comes from letting the system find regularities too numerous and too diffuse for anyone to write down; a representation humans could read would be a representation humans could have written.

AIIA DefinitionDéfinition · black boxboîte noire · fr: boîte noire· en : black box

A system whose internal decision-making cannot be inspected: what goes in and what comes out can both be observed, and the process connecting them cannot be read.Système dont on ignore le mécanisme interne de décision : on observe ce qui y entre et ce qui en sort, sans pouvoir lire le processus qui relie les deux.

NotesRemarques Trained neural networks are the standard case, since their behaviour is stored across millions of connection strengths and no single one of them means anything on its own; how such a system handles a given kind of text is found out by testing it rather than by reading it. The opacity is sometimes a property of the method and sometimes a commercial choice, when a provider withholds the workings of a model it sells.Le cas courant est celui des réseaux de neurones entraînés, dont le comportement tient dans des millions de forces de connexion dont aucune ne signifie quoi que ce soit isolément ; on découvre comment un tel système traite un type de texte en l'essayant, non en le lisant. L'opacité tient parfois à la méthode, parfois à un choix commercial, quand une entreprise garde secret le fonctionnement d'un modèle qu'elle vend.

Other attested French terms: boîte opaqueAutres termes français attestés : boîte opaque

Explainable AI and interpretability

Two overlapping fields work on this problem, explainable AI (accounting for a decision after the fact) and interpretability (how far a model’s workings can be understood directly): domains in which the goal is to develop techniques that show which parts of an input most influenced a decision, or what a particular unit inside a network responds to. It is beyond the scope of the materials in this module, but worth knowing this field exists.

The distinction between the two is itself contested. An account after the fact is not the same as a model whose workings can be read. For high-stakes decisions, Rudin (2019) argues, the answer is to use a model that is interpretable by design rather than to attach an explanation to a black box. Even the best of these techniques leaves a network far less transparent than a rule you can read, or than a statistical model whose weights sit beside named features.

Four consequences follow for anyone who works with these systems:

Errors cannot be fixed the way a rule is fixed. There is no line to correct. Improving a network means changing its training data, training it further, or changing its architecture and training again, and any of these can shift its behaviour elsewhere in ways nobody chose.
Behaviour is discovered by testing, not by inspection. The only way to know how a system handles legal text, or Québécois French, is to try it on legal text and Québécois French, which is why evaluation is a professional skill in its own right (see also the Evaluating language technology module) (voir aussi le module L'évaluation des technologies langagières).
The same input need not give the same output. A classifier is deterministic: the same email gives the same verdict every time. A system that generates text often is not, because choosing each next word can involve randomness, and one run then differs from the next. A single trial is evidence about that run, not about the system.
Confidence is not self-knowledge. A network reports probabilities, and they measure how its training data looked, never how much it “knows” (see also the Critical issues with LLMs module) (voir aussi le module Enjeux critiques des GML).

Key pointsL'essentiel

A large language model is the fourth paradigm: one model, trained once on text with no task in view, that does many tasks once the task is described in a prompt.
A developer used to build the task into the system; now the task is specified at use time, by the person using the model. That moves the line between building a system and using one.

One model, many tasks

Every system of the first three paradigms was built for one task: a spam filter sorts email, a tagger labels words with their grammatical class, and the next system means starting again, with new rules, new features or new labelled data. A large language model is a neural network of the kind this chapter describes, trained from examples like the second and third paradigms, except that the examples are plain text and the task is to predict the next word (Jurafsky & Martin, 2026). Nothing in that training names a task, and once trained, the same model translates, summarizes, lists the terms in a text or labels its words, when the request is written to it in a prompt (Brown et al., 2020). Figure 6figure 6 sets the two arrangements side by side.

Two panels. Left, one system per task: three separate boxes, a tokenizer, a tagger and a term extractor, each with its own text as input. Right, one model, the task in the prompt: three prompts, cut this text into words, label each word with its class, list the terms in this text, all going into a single box, one language model trained once on text with no task in view. — Figure 6. Three systems, each built for its task, and one model that is given each task in a prompt.

Decreasing human influence

Each shift took one thing out of the developer’s hands, as the table below sets out, and the last took the largest: the task itself, which passes to the person using the model.

Shift	No longer specified by the developer	Still specified by people
Rules to machine learning	the patterns	the task and the features, by the developer, before training
Machine learning to deep learning	the features	the task and its data, by the developer, before training
Deep learning to large language models	the task	the task, by the person using the model, in a prompt

The three shifts, and the part of the work the developer no longer specifies after each one.

A large language model is, technically, deep learning. Four things make it a paradigm of its own all the same, rather than a step inside the third:

One general system instead of a system per task. The line is not absolute, since a model can be fine-tuned for one task or one context (see also the Fine-tuning chapter, in the LLM lifecycle module) (voir aussi le chapitre Affinage, dans le module Cycle de vie des GML) and dedicated systems are built on top of models, but its capabilities are far broader than those of any earlier system, and a prompt alone reaches most of them.
A far simpler interface, so far less specialized users. A prompt is ordinary text rather than a program, so the person who describes the job is often the person who used to be a user of someone else’s system. The line between building a system and using one has moved.
An even larger appetite for data and computation, in place of explicit knowledge. Each earlier shift already traded knowledge written by people for data; this one trades the task description itself for more text and more computation.
A blurrier line between natural language processing and the rest of AI. The model that labels words or extracts terms is the same one that writes code, describes images and acts on your behalf, and language has become the interface to all of it.

Calling myself an AI researcher

Before ChatGPT I would never have called myself an AI researcher: I was a researcher in natural language processing, or a computational linguist. Since then, “AI” has come to mean large language models for most people, and a large language model is language technology, the second L stands for language, so when I talk to non-specialists, “AI researcher” is now the simpler description.

A prompt steers a model without changing it, and within limits (see also the In-context learning chapter, in the LLM lifecycle module) (voir aussi le chapitre Apprentissage en contexte, dans le module Cycle de vie des GML).

Still deep learning

Everything that was true of the third paradigm is still true of the fourth:

the patterns it learned are spread over billions of numbers nobody can read (see also the black box page) (voir aussi la page La boîte noire);
everything it does comes from its training text, with that text’s gaps and leanings (see also the Critical issues with LLMs module) (voir aussi le module Enjeux critiques des GML);
the tasks it does well and badly do not follow what a person finds hard, so its abilities have to be tested;
the dedicated systems have not disappeared: for a task run on whole corpora, fast and the same way every time, a trained tagger or a term extractor is still the usual tool (see also the Comparing term extraction methods page, in the Classical NLP module) (voir aussi la page Comparer les méthodes d'extraction de termes, dans le module TAL classique).

Administrator

From neural networks to large language modelsDes réseaux de neurones aux grands modèles de langue