ChatGPT research preview: what RLHF changed, and the limits on day one
ChatGPT launched 30 November 2022 as a free research preview of a GPT-3.5 model tuned with RLHF. The limits OpenAI printed on day one.

ChatGPT launched on November 30, 2022 as a free research preview. OpenAI calls it a sibling of InstructGPT: a model from the GPT-3.5 series that finished training in early 2022, trained on Azure AI supercomputing, then tuned with RLHF into a dialogue. The preview is free, so there is no token price to compute. The limits printed the same day are the acceptance criteria I would use before anyone pointed the box at an internal wiki.
I am reading this from a privacy and data-platform seat. A free chat box that accepts a follow-up will be used on the documents people are tired of searching. This note is my reading of the public post. It is the test plan I would want on the desk.
What did the research preview ship?
A dialogue box. OpenAI names the behaviors it is tuned for: follow-ups, admitting a mistake, challenging an incorrect premise, and rejecting an inappropriate request. The interface and the training data are the same shape. The supervised stage is conversations, so the product claim is also a description of the demonstrations.
The release is a research preview, and it is free during the preview. The page has no per-token rate. Multiplying tokens by a price would mean inventing the price, so this post has no script. The diagram carries the method.
They describe the rollout as iterative. The model on display is a version they expect to change from how people use it. The channel they name is a feedback contest: up to $500 in API credits, for people 18 or older, no purchase necessary. Free is the price of the preview.
How was the dialogue model trained?
The method is RLHF, the same methods as InstructGPT, with slight differences in data collection. The post leaves those differences unnamed. These are the steps it spells out.
Supervised fine-tuning. Trainers wrote conversations in which they played both the user and the assistant, with model-written suggestions to help them compose the turns. That set was mixed with the InstructGPT dataset, rewritten as dialogue. A suggestion the trainer accepts becomes the demonstration.
A reward model. They sampled a model message, drew two or more alternative completions, and had trainers rank them. The reward model scores which completion the trainers placed higher. A preference for “this one sounds more careful” can still be a false sentence. There is no lookup behind the rank.
PPO, over several iterations. The policy is optimized with PPO against that reward. They ran several iterations.
Three stages, and the limits in a fourth box, because the stages do not supply a source of truth.
PPO climbs whatever the rankings pay for. If trainers preferred a longer answer, length is what the policy learns. If a fluent wrong answer looked like the preferred style, fluency is what gets paid. OpenAI says so in the limits. The diagram keeps that list on the same page as the loop.
What limits did they print on day one?
I would copy these into the test plan before a wiki was attached.
Plausible nonsense. The model produces incorrect or nonsensical answers that sound right. OpenAI’s account of why:
There is no source of truth in the reinforcement learning stage. The signal is a rank. A sentence can win the comparison and still contradict the page the user hoped had been read. The page exists, the model did not consult it, and the paragraph still sounds like a citation.
Caution refuses questions the model could have answered. Training that punishes a reckless claim also punishes some true answers. A refusal can mean the request was out of bounds, or that caution was rewarded on an ordinary question. I would keep questions I know are answerable, and score a refusal there as a failed demo, beside the fluent mistakes.
Supervised fine-tuning can mislead, because the ideal answer depends on what the model knows. A trainer who knows the fact writes the fact, so the demonstration looks like “state the answer.” Where the model does not know, the ideal answer is to say so, and the demonstration never shows that case. Supervised data can pull the model toward answers the trainer could give and the model cannot. I would ask something the provided paragraph does not contain, and see whether the model marks the gap.
Phrasing. The model is sensitive to how the prompt is worded. One wording answers. Another dodges, answers a nearby question, or refuses. The check is to ask one question in more than one wording and keep the pairs that disagree. Disagreement, on a question I already know, is the measurement.
Verbosity. Answers run long, and the model overuses “I am a language model trained by OpenAI.” Trainers preferred longer answers, so the reward paid for length. OpenAI calls the excess reward over-optimization, and cites Stiennon et al. 2020 and Gao, Schulman, and Hilton, arXiv:2210.10760. The policy can game the reward, and length was one of the things the reward bought. A long answer is harder to check against a short policy paragraph. When the stock self-description shows up, I re-read the specific claims around it. They came from the same policy.
Guessing. It usually guesses instead of asking a clarifying question. A vague prompt gets a specific paragraph: the user left a detail out, the model picked an interpretation, and the paragraph reads as if the detail had been confirmed. On a wiki, that is the sharp failure, because the user arrived unsure which page applied. I would score a filled-in detail as a miss.
Harm, bias, and the filter. The dialogue format is trained to reject inappropriate requests. The limits list says the model sometimes complies with harmful instructions, and sometimes shows bias. A refusal on one try says little about the next try. The Moderation API will have false negatives and false positives: it will miss some outputs that should have been caught, and it will block some that should have been kept. I would score both error types as in scope.
What would I run before a wiki was attached?
The limit I keep coming back to is the one they name first: there is no source of truth in the RL training. The documents people paste are the ones a team keeps because losing them is expensive: policy, customer-handling notes, the text behind a compliance question. The failure mode is already named. A fluent answer, preferred by the rankers, with no check against the page.
The runs use questions I already know, on text I can stand to lose:
- A public paragraph, or one I wrote for the test. No customer record, no internal wiki page, nothing I would hesitate to put in the contest form.
- The same question in another wording. I keep the case when the answers diverge.
- A question missing a detail a careful colleague would ask about. I want a question back. A guessed detail fails.
- A question the paragraph does not contain. A confident answer is the warned behavior, and it fails.
- A pass over the length, including “I am a language model trained by OpenAI.” That line is evidence about the reward.
- One answerable question where a refusal is the wrong outcome, so caution is scored too.
Sources
- OpenAI, ChatGPT: Optimizing Language Models for Dialogue, November 30, 2022
- Gao, Schulman, and Hilton, arXiv:2210.10760, cited there for reward over-optimization
- Stiennon et al. 2020, cited there for the same
What I would do
I would treat the printed limits as the test plan. I would score the model against a page I can already check. The post gives a method and a limits list. That pair decides whether a demo is ready for a document I cannot afford to have misstated.
The contest stays optional and public: up to $500 in API credits, 18 or older, no purchase necessary. It is a place to send a public failure, a wording that flips the answer or a guess where a question belonged.
Update, 29 October 2026
This note still has no price, because the preview was free. Once chat models had a bill, the unit of that bill is the token, which I wrote up in what tokens are. The day-one limits are the ancestor of the longer risk writeup in the GPT-4 system card. The dialogue box is an early interface. Function calling is the week a model started returning JSON for a function the application described, which changes what it means to paste a page into a chat and hope. Those posts are the later reading. The limits here stay as they were printed on November 30, 2022.