A language model is a system that predicts the next token in a sequence of text. When the model is large enough and trained on enough data, the same simple objective produces a model that can answer questions, write code, translate between languages and summarize long documents.
A foundation model is a large model that can be adapted to many different tasks. Instead of training a new model for every task, engineers start from a foundation model and adapt it with prompts, retrieval or finetuning.
The model does not read words. It reads tokens. A token can be a whole word, a part of a word or a single character. The tokenizer decides how text is split into tokens, and the model only ever sees the resulting numbers.
The cost of using a model depends on the number of tokens. Every token in the prompt and every token in the response is counted, and the provider charges for each of them. Latency also grows with the number of tokens the model has to generate.
Training data shapes everything the model can do. If a language or a domain is rare in the training data, the model will be weaker in that language or domain, and its tokenizer will split that text into more tokens.
Most training data comes from the public web. The web is large, but it is not clean, and it is dominated by English. Teams filter the data with simple rules, but the filtered data still reflects what is common on the internet rather than what a specific product needs.
After pre-training, the model is good at completing text but not at following instructions. Supervised finetuning teaches the model to answer questions by showing it examples of good answers written by people.
Preference finetuning goes one step further. People compare two answers and choose the better one, and a reward model learns from these comparisons. The language model is then trained to produce answers that the reward model scores highly.
Sampling is how the model chooses the next token. The model computes a score for every token in its vocabulary, turns the scores into probabilities and then picks one token. A low temperature makes the choice more predictable. A high temperature makes it more creative and less consistent.
Because sampling is random, the same question can receive different answers. This is useful for creative work and painful for tasks that need consistent answers. Engineers fix the temperature, fix the random seed and cache answers to reduce this problem.
A model can also produce answers that sound confident but are not true. This is called hallucination. Short answers, clear instructions and grounding the answer in retrieved documents all help reduce it.
Evaluation is the hardest part of building with foundation models. The answers are open ended, so there is rarely a single correct output to compare against. Good teams build an evaluation pipeline before they build the product.
A prompt is the instruction and context given to the model. A good prompt is clear, specific and tested on real examples. Prompts should be versioned like code, because a small change in wording can change the output.
Retrieval augmented generation gives the model access to documents it has not seen during training. The system searches for relevant passages and adds them to the prompt, so the model can answer from the documents instead of from memory.
An agent is a model that can use tools. It plans a sequence of steps, calls a tool, reads the result and decides what to do next. Agents are powerful, but every step is another chance for an error.
