What is constitutional AI? Anthropic's RLAIF method, explained
Constitutional AI trains a model against written principles using AI feedback instead of human harm labels. How critique, revise and RLAIF work, in Python.
Tags
11 posts tagged
Constitutional AI trains a model against written principles using AI feedback instead of human harm labels. How critique, revise and RLAIF work, in Python.
RLVR trains an LLM with rewards from a program that checks the answer, such as a math checker or unit tests, instead of a reward model. How it works, in Python.
Model distillation trains a small student model to copy a large teacher's outputs, keeping much of its skill at a fraction of the cost. How it works, in Python.
Scaling laws predict how an LLM's loss falls as you add parameters, data and compute. What Kaplan and Chinchilla found, and why models now overtrain.
Synthetic data is training data written by a model, not people. How LLM labs generate and verify it, why model collapse happens, with a Python demo.
DPO vs RLHF: DPO trains on preference pairs with one classification loss, no reward model or PPO loop. How the two compare, with a Python demo that checks it.
LoRA fine-tunes an LLM by training two small low-rank matrices per layer while the model stays frozen. QLoRA does it on a 4-bit base. How both work, in Python.
Fine-tuning an LLM means training a pretrained model further on your own examples so its weights change. How SFT works, what it costs, and what it forgets.
How LLMs are trained: pretraining on trillions of tokens, then supervised fine-tuning, RLHF or DPO, and RL with verifiable rewards. With a compute calculator.
A transformer is the neural network architecture behind modern LLMs. How its layers, attention and feed-forward blocks work, with a Python parameter counter.
RLHF trains a reward model on human preference rankings, then tunes an LLM to score well on it without drifting far. The three steps, with a Python demo.