Mistral Large 4 hardware: what self-hosting takes
Mistral Large 4's open weights are 1,060.6 GB in FP8 and 610 GB in NVFP4. What fits on one 8-GPU node, KV cache per token, and API vs renting GPUs.
Tags
25 posts tagged
Mistral Large 4's open weights are 1,060.6 GB in FP8 and 610 GB in NVFP4. What fits on one 8-GPU node, KV cache per token, and API vs renting GPUs.
A safetensors file starts with an 8-byte header length and a JSON header. Two Range requests per shard counted Mistral Large 4's 1,053B params from 6.2 MB.
Gemini 4 Carbon will cost $4/$20 per million tokens after its intro price, the same as Opus 5.5. Its implicit-only cache decides which is cheaper.
A KV cache pool smaller than the context window made Cloudflare's GLM-5.3 return other users' data in 155 responses. How it happened and the SGLang fix.
Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang (Kevin Flansburg and Michelle Chen, Cloudflare, October 26, 202...
via Hacker NewsUbicloud took Kimi K2.6 tool-call schema accuracy in vLLM from 92% to 100% with constrained decoding. What it does and why vLLM leaves it off by default.
Fixing Tool-Call Accuracy in vLLM (Jesse Donkervliet and Junhao Li, Ubicloud, October 20, 2026) Ubicloud serves open-weight models, and this post explains why a...
via Hacker NewsXGrammar cuts the next-token menu to 2 to 12 tokens on JSON structure. With whitespace off it still expects ': ' after keys unless you pass separators.
"It's a simple question for any enterprise. Would you like something that's 10 percent of the cost if it's 90 percent as good as the best, or do you want to pay that 10 times higher cost?"
— Arvind Krishna
An agent task's time is generation, prefill, tool runs and per-call overhead. A runnable model shows what 1000 tokens per second fixes and what it doesn't.
Prompt caching lets an LLM API reuse work on a prompt prefix it has already processed, cutting input cost and latency. How it works, prices, and a Python demo.
Quantization stores model weights in fewer bits plus a scale. Runnable int8, int4, MXFP4 and NVFP4 code, the memory math, and where accuracy goes.
Sparse attention lets each token attend to a few chosen earlier tokens. How DeepSeek V3.2 does it, why long prompts get cheaper, and a runnable numpy demo.
Claude Opus 4.5 costs $5/$25 per million tokens and adds a beta effort setting. How effort changes token spend and cost per task, with a runnable script.
A context window is how many tokens a model handles per request. Gemini 3 Pro takes 1M, but past 200k each token costs double. Runnable cost script inside.
Reasoning models write hidden thinking tokens before answering, billed as output. How GPT-5.1's reasoning effort works, with a runnable cost script.
DeepSeek released R1 on 20 January 2025 under MIT, at $0.55 per million input tokens, with 79.8% on AIME 2024. What that does to a reasoning-search bill.
LLM latency is time to first token (TTFT) plus output tokens times inter-token latency. What each metric measures, what moves it, with a Python model.
How much VRAM an LLM needs: parameters times bytes per weight, plus the KV cache for your context. Worked numbers for 8B to 405B models, in a Python script.
Test-time compute is the work a model does while answering: longer reasoning, more samples, a verifier. How it scales and what it costs, with Python.
o1-preview launched 12 September 2024. Thinking is billed as output, the chain of thought stays hidden, and the IMO score in the post is not the preview.
FlashAttention computes exact attention in tiles that stay in fast GPU SRAM, so the N x N score matrix never hits HBM. How tiling works, with Python.
LLM routing sends each query to the cheapest model likely to answer it well; cascades try a small model first and escalate. How both work, with a Python sim.
Speculative decoding has a small draft model guess several tokens and the big model verify them in one pass. Why the output stays identical, with Python.
Mixture of experts (MoE) routes each token to a few expert networks, so a model can hold 47B parameters but compute with 13B. How routing works, in Python.
The KV cache stores each token's attention keys and values so an LLM doesn't recompute them per new token. How it works, how big it gets, and GQA.
vLLM is an open-source LLM serving engine built on PagedAttention, which stores the KV cache in small blocks like OS pages. How it works, with a Python demo.