Running an open-weight model on your own machine often produces weaker answers than the hosted version of the same model. The gap usually comes from quantisation, context limits, sampling settings and prompt scaffolding rather than the weights themselves.
Key takeaways
- A model run locally and the same model served by a provider can behave differently because the surrounding software stack, not the weights, differs in several ways at once.
- Quantisation shrinks a model so it fits in consumer memory, and the compression that makes local inference possible can also reduce accuracy on harder tasks.
- Hosted services typically apply their own system prompts, sampling defaults and formatting layers, so users are rarely comparing like with like.
- Short context windows and aggressive context truncation on local setups can silently drop earlier parts of a conversation, which reads to the user as forgetfulness.
- Most of the perceived gap is diagnosable by checking configuration first, though some quality loss from heavy compression on modest hardware is genuine and not a settings error.
What is actually happening when a local model underperforms
A large language model is a set of numerical weights, but running it requires a great deal of machinery around those weights: a runtime that loads them into memory, a tokeniser that turns text into model inputs, a sampler that chooses each output token, a template that formats the conversation, and a context manager that decides what history the model sees. Two systems can use identical weights and still produce noticeably different output if any of those layers differ.
When people say a local model feels less capable than the same model accessed through a hosted interface, they are usually observing the sum of those differences. The model may be compressed to a smaller numerical format, the conversation template may not match what the model was trained on, sampling parameters may be set to values that encourage rambling or repetition, and the effective context may be far shorter than the model nominally supports. Each factor on its own can be modest. Together they can be the difference between a coherent answer and a confused one.
Why this is being discussed now
Open-weight models have become far easier to run on ordinary hardware. Desktop applications, command-line runtimes and packaging formats have removed most of the friction that once limited local inference to people comfortable compiling code. As the audience widens, more users are running the same named model in two places and comparing the results directly — something that was rare when local inference was a specialist activity.
The result is a recurring discussion in developer communities about why the local copy seems worse. The discussion recurs because the causes are not visible from the interface. A user sees a model name and a chat box; they do not see the quantisation level, the chat template, the temperature or the context truncation policy. Attention has shifted towards explaining those layers rather than towards any single new product or release.
The background a newcomer needs
Model weights are stored as numbers with a certain precision. Training and reference inference commonly use higher-precision formats, which consume substantial memory — enough that large models do not fit on a typical consumer graphics card. Quantisation reduces the precision of those numbers so the model occupies less memory and runs faster. This is what makes local inference practical at all.
Quantisation is a trade-off rather than a free saving. Milder compression tends to preserve behaviour closely; more aggressive compression saves more memory and generally costs more quality. Where the loss becomes noticeable depends on the model, the quantisation method and the task, and it is not uniform: effects often show up first on long multi-step reasoning, precise arithmetic or code, and less on casual conversation. Published comparisons exist, but results vary by method and by benchmark, so there is no single universally applicable threshold.
Two further pieces of background matter. First, instruction-tuned models expect their conversation to be formatted with specific control tokens marking system, user and assistant turns; if the runtime applies the wrong template, the model receives input unlike anything it was trained on. Second, the advertised context window is an upper bound, not a guarantee — running at full context requires memory for the key-value cache, and local setups frequently configure a much smaller window to fit available hardware.
Who is affected and how
Individual users running models on laptops and desktops feel this most directly. Their hardware constrains which quantisation they can use and how much context they can afford, so they are pushed towards the settings most likely to degrade output. They are also the least likely to have instrumentation that would reveal what is happening.
Developers building on open-weight models face a different version of the problem. If prototyping happens against a hosted endpoint and deployment happens on self-hosted infrastructure, behaviour can shift between the two environments even with the same nominal model, and prompts tuned in one setting may not transfer cleanly.
Organisations choosing between hosted and self-hosted deployment are affected at the level of procurement. Self-hosting is often motivated by data control, cost predictability or offline operation. If the resulting quality gap is attributed to the model rather than to the deployment configuration, an organisation may draw the wrong conclusion about whether open-weight models suit its needs.
Where informed people disagree
There is genuine disagreement about how much quantisation actually costs in practice. Some argue that moderate compression is close to lossless for most everyday use and that reported degradation is mostly attributable to misconfiguration. Others argue that the damage is real but concentrated in exactly the capabilities people test with — multi-step reasoning, long-context recall, precise instruction following — so aggregate benchmark scores understate it.
A second disagreement concerns benchmarks themselves. Standard evaluations may not capture the failure modes users notice, and small differences in evaluation setup can change results, which makes cross-comparison difficult. A third concerns responsibility: some hold that runtimes should ship conservative defaults that favour correctness, while others prefer defaults that maximise speed and let users tune upwards, on the grounds that most local users prioritise responsiveness.
Underlying all of this is disagreement about how much of the hosted-model experience comes from the model at all, as opposed to the retrieval, tool use, system prompting and post-processing wrapped around it. That share is not publicly documented for commercial services.
What this means in practice
The practical implication is that diagnosis should precede conclusions. Before deciding a model is inadequate, it is worth establishing which quantisation is loaded, what context length the runtime is actually configured for, whether the chat template matches the model’s expected format, and what sampling parameters are in use. These are usually visible in a runtime’s configuration or startup output.
It also means comparisons should be made deliberately. Testing a heavily compressed local copy against a hosted service with its own prompt scaffolding is not a comparison of models. A fairer test uses the same prompt, comparable sampling settings and a context window long enough for the task.
Finally, it means hardware constraints are real constraints. If memory only permits aggressive quantisation of a large model, a smaller model at milder compression may perform better on the same machine. That trade-off is specific to the hardware, the model family and the intended task, and generally has to be tested rather than assumed.
What to watch next
Several developments would change this picture. Quantisation methods continue to improve, and techniques that better preserve quality at a given memory footprint would narrow the gap. Clearer disclosure by runtimes — showing quantisation level, effective context and template in the interface rather than in logs — would make the causes visible to non-specialist users.
Evaluation practice is also worth watching. Comparisons that test quantised variants under stated conditions, rather than reporting a single score per model name, would let users make informed choices. Hardware trends matter too: more memory on consumer devices reduces the pressure towards aggressive compression.
Whether the perceived gap narrows will depend less on new model releases than on whether the layers around the weights become easier to inspect and configure correctly.
Frequently asked questions
Does quantisation always make a model worse?
Not always noticeably. Milder quantisation typically preserves behaviour closely enough that most users will not detect a difference in ordinary conversation. More aggressive compression generally costs more, and the loss tends to appear first on demanding tasks such as multi-step reasoning, precise arithmetic and code generation. How much is lost depends on the model, the compression method and what is being asked, so results vary considerably between setups.
Why does my local model forget earlier parts of the conversation?
The most common cause is a context window configured smaller than the model supports. Full context requires memory for the key-value cache, so local runtimes often default to a shorter window to fit available hardware. When a conversation exceeds it, older content is dropped or summarised, which reads as forgetfulness. Checking and raising the configured context length, if memory allows, usually addresses this.
Is the hosted version of an open-weight model the same model?
The weights may be the same, but the surrounding system is generally not. Hosted services typically add their own system prompts, sampling defaults, formatting and sometimes retrieval or tool use. They may also serve at higher precision than a consumer machine allows. Providers do not usually document these details fully, so the extent of the difference for any particular service is not publicly known.
What settings should I check first on a local setup?
Start with four: the quantisation level of the weights you loaded, the context length the runtime is configured for, whether the chat template matches the model’s expected conversation format, and the sampling parameters such as temperature. Most runtimes report these at startup or in a settings panel. Mismatched templates and unexpectedly short contexts are frequent and produce failures that look like poor model quality.
Is a smaller model better than a heavily compressed large one?
Sometimes, but it depends on the specific pair and the task. A smaller model at mild compression may outperform a much larger model compressed aggressively enough to fit the same memory, and the reverse also occurs. There is no general rule that holds across model families and workloads, so the practical approach is to test both options on representative tasks with your own hardware.
Does running locally mean my data stays on my machine?
Inference performed entirely on local hardware does not send prompts to an external service. However, some applications combine local inference with online components such as search, model downloads or telemetry, and configurations vary between tools. Whether a particular setup keeps everything local depends on that tool’s design and settings, which should be checked in its documentation rather than assumed from the term “local”.
Sources and further reading
- Technical documentation from open-source inference runtimes, which describes quantisation formats, context configuration and chat templating.
- Model cards published by organisations releasing open-weight models, which state expected prompt formats and context limits.
- Academic literature on neural network quantisation, covering methods for reducing numerical precision and their measured effects on accuracy.
- Developer community discussion forums, where practitioners compare local and hosted behaviour and report configuration issues.
Surfaced from the hackernews signal “local model quality gap”. AI-assisted draft, editorially reviewed.
