What happens to the training textCe qu'il advient du texte d'entraînement
TRA6008 To read for S3À lire pour la S3
Nothing left to consult
Once pretraining ends, the corpus is gone: not shipped with the model, not consultable, no article to reread and no document to retrieve. Everything it has is in its parameters, as numbers.
The corpus also stopped somewhere. Whatever text went into it was gathered up to a point in time, and nothing after that reached the model during training.
Systems that do search
Many systems you use do search. A chat assistant that browses the web, or that answers from documents you upload, is running a search and putting the results into its context before it answers. That search is a separate component around the model, and the retrieved text arrives the way your own prompt does.
The promptLa requête
I want to generate a figure to illustrate how an LLM, without access to tools, might be compared to a human who has read a lot but no longer has access to all of the books they have read: you hope they are able to give a reasonable answer, but they cannot ground it in the resources. The point is to make clear that LLMs cannot look something up in a database: they construct from learned connections. Versus an LLM with tool use is like that same person, but now also with access to a library and internet and a calculator etc. to check its answers. Draft in a simple and clean style, without border lines and with soft colours. This is for academic purposes, so do not make it cutesy or spectacular: focus on accurate and illustrative.
Learned parameters, not stored data
The corpus is orders of magnitude bigger than the model, so its parameters keep patterns and associations rather than the text itself (see also the LLMs as next-word predictors chapter) (voir aussi le chapitre Les GML, prédicteurs du mot suivant).
Verbatim reproduction does happen all the same. Text repeated many times across the corpus can be learned closely enough to come back out word for word, and Nasr et al. (2025) showed that a deliberate attack can make a production model emit training data at a far higher rate than ordinary use produces.
Plus rien à consulter
Une fois le préentraînement terminé, le corpus a disparu : il n’est pas livré avec le modèle, il n’est pas consultable, il n’y a aucun article à relire ni aucun document à récupérer. Tout ce que le modèle possède est dans ses paramètres, sous forme de nombres.
Le corpus s’est aussi arrêté quelque part. Le texte qui y est entré a été rassemblé jusqu’à un moment donné, et rien de postérieur n’est parvenu au modèle pendant l’entraînement.
Les systèmes qui font des recherches
Bien des systèmes que vous utilisez font, eux, des recherches. Un assistant conversationnel qui navigue sur le Web, ou qui répond à partir de documents que vous téléversez, lance une recherche et met les résultats dans son contexte avant de répondre. Cette recherche est un composant distinct, autour du modèle, et le texte récupéré lui parvient comme votre propre requête.
The promptLa requête
I want to generate a figure to illustrate how an LLM, without access to tools, might be compared to a human who has read a lot but no longer has access to all of the books they have read: you hope they are able to give a reasonable answer, but they cannot ground it in the resources. The point is to make clear that LLMs cannot look something up in a database: they construct from learned connections. Versus an LLM with tool use is like that same person, but now also with access to a library and internet and a calculator etc. to check its answers. Draft in a simple and clean style, without border lines and with soft colours. This is for academic purposes, so do not make it cutesy or spectacular: focus on accurate and illustrative.
Des paramètres appris, pas des données stockées
Le corpus est de plusieurs ordres de grandeur plus gros que le modèle, si bien que ses paramètres gardent des régularités et des associations plutôt que le texte lui-même (see also the LLMs as next-word predictors chapter) (voir aussi le chapitre Les GML, prédicteurs du mot suivant).
La reproduction mot pour mot se produit tout de même. Un texte répété de nombreuses fois dans le corpus peut être appris d’assez près pour ressortir mot pour mot, et Nasr et al. (2025) ont montré qu’une attaque délibérée peut amener un modèle en production à émettre des données d’entraînement à un taux bien plus élevé que l’usage ordinaire n’en produit.
Sources
- Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models. Online draft, 3rd edition. https://web.stanford.edu/~jurafsky/slp3/
- Nasr, M., Rando, J., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., et al. (2025). Scalable Extraction of Training Data from Aligned, Production Language Models. Proceedings of the Thirteenth International Conference on Learning Representations (ICLR 2025). https://proceedings.iclr.cc/paper_files/paper/2025/hash/cce0e917b050208170151f77b497fc71-Abstract-Conference.html
Sources
- Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models. Online draft, 3rd edition. https://web.stanford.edu/~jurafsky/slp3/
- Nasr, M., Rando, J., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., et al. (2025). Scalable Extraction of Training Data from Aligned, Production Language Models. Proceedings of the Thirteenth International Conference on Learning Representations (ICLR 2025). https://proceedings.iclr.cc/paper_files/paper/2025/hash/cce0e917b050208170151f77b497fc71-Abstract-Conference.html