Models and runtimes

If you turn the request “Choose a Korean model that answers questions about internal documents well” directly into a model family search, the list of candidates grows without end. First write down the material users provide, the expected result, errors that cannot be tolerated, and the points where a person checks the output. Document Q&A chains together an embedding model that turns questions into vectors, a retriever that finds relevant documents, a reranker that reorders candidates, and a generative model that writes sentences based on the evidence. If tables are images, an OCR or Vision stage is added in front. These stages can run on different models and runtimes, and if you label a failure in one stage as a lack of overall LLM capability, you cannot find the cause.

Set an output contract for each role. For a generative model, what matters is correct answers·evidence citations·JSON format·holding when it does not know; for embedding, recall of the correct document within top-k; for a reranker, the rank of the correct document and added latency; for OCR, field·character·number accuracy; and for Speech-to-Text, word error rate and proper-noun accuracy. Merging them into a single “90% accuracy” column creates a false comparison of metrics whose denominators and failure meanings differ. Keep each role's metric formula, judge, and minimum sample size together in the catalog schema.

Create normal, boundary and failure examples before inspecting candidates. Generation includes unsupported questions, conflicting documents, dates, amounts and structured outputs as well as common questions. Retrieval includes short titles, long tables, synonyms and unanswerable questions. Vision includes blurry photos, rotation, small Korean text and merged table cells; speech includes background noise and internal abbreviations. Without a gold set reflecting real-user frequency and harm, only public benchmarks and demo impressions remain.

Text/image input or multilingual support does not automatically transfer a pass between roles. Image input does not guarantee numerical field accuracy on Korean receipts, and long context does not guarantee citing a clause on the final page. A small specialist reranker or OCR model may be more accurate and faster at its stage than a large generative model. Link the final system to each stage’s passing results, and ensure replacements can first be regression-tested on that role’s same workload.

Why does this happen?

This is because separating functions with different inputs, outputs, and error costs lets you find only the models you need and replace only the failed stage.

When is it a problem?

Choosing every capability from one popularity ranking can misdiagnose retrieval-recall deterioration as a generation-model problem, leaving errors even after purchasing a larger model.

Common beginner misconceptions

Multimodal or multilingual labels are not certification marks guaranteeing the same quality across every image·language·task.

How to verify it yourself

Draw your workflow as stages, and for each stage write one line each for input, output, gold answer, required metrics, and human approval points.

On the Hub, community uploads, merges, adapters, and conversions with names similar to the official creator's appear alongside the originals. Do not pick the original based only on download counts or names in search results. Cross-check that the organization and repository are the ones the creator's official site, GitHub repository, or model card points to. First confirm where release-specific model cards and licenses live, as in Meta's Llama model repository, and where variants of a model family are described, as in Google's official Gemma documentation. Treat a re-upload without an official link as a separate candidate of unknown origin.

A catalog identifier comprises organization/repository, exact commit revision, downloaded file names and bytes, and cryptographic hashes, not a display name. Branches and moving references such as `main` or `latest` may change after review, so pin the evaluation commit. Record the download date without using it as a substitute for revision. If the same model ID downloads to a different hash, compare changed files, cards, and terms instead of reusing prior approval unchanged.

Treat configuration, tokenizer, chat template, processor, and special-token files as one bundle with the weights. Mixing old tokenizers with new weights or another family’s role template with an instruct variant can change termination, system-role behavior, and quality. Compare rendered prompts and processor outputs immediately before model calls against official examples, and record each component’s source revision and hash in the manifest. An allowlist of files actually needed for deployment is easier to review than a note that the whole repository was copied.

To confirm reproducibility, another person must be able to download the same artifact again using only the catalog row. If there is an offline cache, record the relationship between the remote source and the local object hash, and for gated models that require access approval, record who downloaded them with which account and terms version. If the artifact cannot be downloaded again because a repository was deleted or permissions expired, keep the approved artifact in a permitted internal registry with retention and access policies. Even if the source disappears, you must be able to prove what was evaluated.

Why does this happen?

Uploads with the same name and moving tags can point to different files, so immutable identifiers are needed to reproduce the causes of results and the security state.

When is it a problem?

Recording only the repository name makes it impossible to tell whether the model, tokenizer, or template changed when quality shifts after an update.

Common beginner misconceptions

High download counts or an official family name in a community repository do not establish that it is the creator’s original source.

How to verify it yourself

Go from the official documentation to the repository, record the organization, commit, file list, bytes, and hashes, and download again in another environment to confirm the same hashes.

Even within the same family, base, pretrained, instruction-tuned, coding, safety, and vision variants differ in purpose and input format. A base model is the starting point of next-token training, while chat·instruct models may assume specific instruction tuning and templates. Do not interpret a name suffix by common sense; check the intended use, architecture, and example inputs defined by the creator. Also determine whether a repository card describes the whole family or the exact checkpoint, so you do not copy conditions from a different size.

Do not combine parameter count, artifact bytes, and runtime memory in one column. Dense and Mixture-of-Experts models can differ in total versus per-token activated parameters, and inclusion of vision encoders·embeddings in name counts varies. Check config layers·hidden size·attention structure and file lists, recording actual bytes by precision and load·prefill·decode peaks. Do not equate “30B-A3B” with a 3B file or 30B dense compute; connect official definitions to runtime logs.

Context windows and modalities are conditional architecture capabilities. Even within one family, official context or image-input support can vary by size, so read the exact variant card. A maximum context figure does not guarantee recall at every position, or concurrency, latency, and quality on your GPU. Ask for evidence at the beginning, middle, and end of minimal, representative, and longest documents to check truncation and lost-in-the-middle effects, and for Vision, also fix resolution, image count, and processor settings.

Do not push the limitations and risks section behind the performance tables. Map the training data overview, supported languages, factuality, bias, and safety, and unevaluated domains to your own usage conditions. If a card has no evidence for Korean or for domains such as medicine or law, do not assume support; record it as a gap requiring local evaluation and expert review. Reflect any "not recommended" or human-supervision conditions in app permissions and approval flows. An empty document is not a pass; it is unconfirmed.

Why does this happen?

Variants within a family differ in input, template, context, and evaluation scope, so you must know the exact checkpoint conditions to compare candidates correctly.

When is it a problem?

Copying the description of a different size or base variant can lead you to expect unsupported image or context capabilities, or ruin quality with the wrong template.

Common beginner misconceptions

The maximum context and public benchmark scores in a model card do not guarantee quality or speed for your languages·documents·hardware.

How to verify it yourself

Extract the variant, modality, context, language, and limitations from the exact checkpoint card and config, and link direct evidence or a gap to each row of your requirements.

Converting Hugging Face's original weights to GGUF for llama.cpp, MLX format, or a runtime-specific quantization produces a new artifact. If you record only a source model ID and save `model-q4.gguf`, there is no way to know which revision, quantization method, and tool version were used. The build manifest records source file hashes, converter and quantizer commits, the command and options, calibration data identifiers, and output file names, sizes, and hashes. Container images and build architecture also affect the result, so preserve the build as a rerunnable recipe where possible.

Community-made quantized files are convenient, but their provenance gap must be evaluated separately. Check whether the card includes the source revision and conversion command, whether the tokenizer and template match the original, and whether all split files are complete. The same Q4 label can differ in quality and kernel support depending on method, group, and mixed precision. Do not treat the quant label in a file name as a guarantee of performance or quality; read the actual metadata and conversion records.

Artifact security involves more than malicious code. Pickle-based files can execute code during deserialization, and custom modeling code or `trust_remote_code` executes repository code. Check Safetensors usage, required file-type allowlists, malware and secret scans, and dependencies. Fetch even official-organization files with least privilege in isolation and verify hashes. Check that the card viewed in the browser corresponds to the object received by the execution node.

Evaluation and rollback complete the lineage. Link each artifact’s role-specific evaluations and runtime and driver versions to failing inputs, logs, and the previous approved artifact’s hash. Put a smaller quantized candidate on hold if numerical extraction, Korean subset quality, or tool formatting regresses. Reconvert from the source or return to the previous artifact, and verify recovery of the same failing inputs before narrowing the cause to the quantization change.

Why does this happen?

Conversion and quantization change bytes, computation and errors, creating artifacts independent of the original that need quality, security and compatibility evidence.

When is it a problem?

Without a source revision and recipe, you cannot reproduce a quality regression in a smaller file or safely recreate it.

Common beginner misconceptions

Sharing a family and Q4 name does not make two models bit-for-bit identical regardless of who made them or how they were converted.

How to verify it yourself

Build a chain from the source hash through tool, command, and calibration to the output hash, and regenerate identical results in an isolated clean build.

Open-weight usually denotes the distribution fact that trained weights are accessible. This alone does not establish that training data, source code, and training recipes are all public or that a standard open-source software license applies. In the catalog, keep weight access, model license, code license, and data·recipe disclosure scope in separate columns. Replace the marketing phrase “open model” with concrete statements about which files can be obtained and what rights you have.

Read the exact release's license file and acceptable use policy. Check whether commercial use, redistribution, derivative works, attribution and notices, monthly-user or organization-size conditions, prohibited domains and geographic restrictions apply to your deployment. The Hub license metadata tag helps discovery but does not replace the original license and accompanying policies. Record the URL, file hash or revision, review date and reviewer, and refer important interpretation questions to your organization's legal review.

Adapter training, merging, quantization, and API delivery also raise different questions. Determine whether the obligations of the base model and the dataset and code licenses conflict, whether converted artifacts can be delivered to customers, and how to provide attribution, the card, and notices. Serving only through a server API without distributing files does not automatically remove prohibited-use, privacy, and output responsibilities. Do not cover the terms of use for an internal pilot and an external product under the same approval.

If the license is unclear, the candidate is on hold regardless of technical approval, even with high benchmarks. The person responsible must not fill blanks with a guessed "permitted"; instead, they record the questions to confirm, the advice needed, and alternative candidates. When the model release changes, review the diffs of terms and policies as well as weights. Reuse a previous version's review only where conditions are the same, and if new obligations arise, update deployment notices, access controls, and catalog status together.

Why does this happen?

Even a technically runnable model is not a lawful, sustainable candidate if its terms of use and redistribution do not fit the task.

When is it a problem?

Approving based only on a short license tag can mean discovering external distribution, derivative artifact, or usage-scale conditions too late and having to withdraw the product.

Common beginner misconceptions

Being able to download weights for free does not grant unrestricted commercial use or redistribution rights.

How to verify it yourself

Tabulate your use, modification, distribution, and prohibited terms from the exact revision's license and policy, and link the responsible reviewer's date and conclusion to the evidence.

A compatibility row records model architecture and artifact format, hardware, operating system and architecture, driver, and runtime and backend versions together. A GGUF loading in llama.cpp does not mean the same file is supported in vLLM or MLX, and the existence of a Hugging Face repository does not guarantee a specific GPU quant kernel. Check the runtime's official supported-model and feature documentation and release notes, and pin the exact combination. Classify combinations missing from the support table as experimental rather than “probably works.”

Chat templates, tokenizers, and processors are part of functional compatibility. Compare the role and special tokens an instruct model expects, vision image processors, embedding pooling and normalization, and reranker input-pair formats against official examples. Test whether system roles, stop tokens, tool schemas, and structured output actually take effect, not merely whether a prompt produces output. Include deliberately invalid values and boundary inputs to detect wrappers silently ignoring unknown options.

Verify the actual execution path in startup logs and the profiler. Record whether every layer or operator runs on the intended GPU or NPU, the amount of CPU fallback and host-device transfer, and whether compiled graphs change as context and batch sizes increase. High utilization on one device does not prove acceleration of the entire graph. Break down latency and peak VRAM and RAM across loading, prefill, decode, embedding, and reranking to distinguish host tokenization, transfer, unsupported kernels, and queue bottlenecks.

Expand from a small passing baseline. Verify exact outputs with short text, concurrency 1 and default precision, then increase context, images, tools, batch and concurrency individually. On failure, revert the last change and check recovery of the same input rather than changing model, runtime, driver and template together. Catalog unsupported features, limitations, alternative paths and version-specific reproduced commands/logs as well as supported features.

Why does this happen?

This is because even the same artifact can run on a different device and differ in features·speed and results depending on the runtime·backend and template support.

When is it a problem?

Recording only load success misses problems in real operation, such as the tool format being ignored or p95 spiking because of CPU fallback.

Common beginner misconceptions

Architecture support in a model card does not automatically cover every quant, hardware, OS, and serving feature.

How to verify it yourself

Pin the exact stack in a manifest, check device placement and output on a short baseline, and then expand features and load one item at a time.

Evaluation sets must represent operational distributions and failure impact. For generative Q&A, define answers and supporting paragraphs from approved documents, with “no evidence” as the correct handling for unanswerable questions. Include numbers, dates, conflicting documents, long tables and prompt injection rather than favoring simple general-knowledge questions. Use compilation, tests and prohibited-API checks for coding, field-level ground truth for vision/OCR, and human-reviewed transcripts for speech. De-identify personal data through approved procedures.

Metrics reflect each role and the failures users experience. For generation, separate correctness, evidence fidelity, citation presence, schema compliance and safe refusals. For embeddings use recall@k; for rerankers use MRR, nDCG and added p95 latency. Do not let overall OCR character accuracy hide amount or account-field errors: require exact matches for critical fields. Likewise, give product-name and number subsets separate STT gates alongside average word error.

Compare the baseline and candidate under the same model input, prompt and template, output limit, sampling, evaluation code, and runtime conditions. Record any template differences the candidate requires, but keep the shared task inputs and judging criteria unchanged. Repeat at least three times to observe stochastic output and system variation, and if an automated judge is used, first validate its agreement with human gold labels and its biases. Do not lower the pass line after seeing results; preserve the predefined thresholds and change history.

Hold the candidate if an important subset fails, even when the overall average passes. Do not let 99% on general English questions conceal 70% on Korean number extraction. Preserve failing inputs, actual outputs, expected answers, judgment reasons and model/artifact hashes. Classify root causes as model, prompt/template, retrieval, quantization or runtime. After fixing one item, rerun the same failure set and the full regression suite to verify that the local improvement has not damaged other quality dimensions.

Why does this happen?

Task-specific errors and harms do not show up in public benchmark averages, so the actual usage distribution and important subsets must be the basis for approval.

When is it a problem?

Overall averages can let numerous easy cases hide severe regressions on essential cases such as missing evidence, numbers and Korean.

Common beginner misconceptions

A larger model or the top public ranking does not necessarily keep the same rank with your prompts, language, and quantization.

How to verify it yourself

Pin role-specific normal, boundary and failure gold sets and predefined thresholds, then compare baseline and candidate results including all failed rows.

In the performance manifest, record the exact model revision and artifact, tokenizer and template, quant, runtime and driver, input and output tokens, sampling, batch and concurrency, and hardware power conditions. Separate cold download, load, and compile, warm Time To First Token, prefill and decode token/s, and end-to-end latency. Record p50, p95, maximum, and error rate instead of a single average, and repeat at least 3 times. If user requests mix short and long documents, include that ratio and the queue in the load.

Memory means peak usage at each stage, not file bytes. Record weights, KV cache or embedding batches, runtime workspace, host RAM·VRAM, and safety headroom. Gradually increase context·image count and concurrency to their targets to find OOM·fragmentation·swap. Even if lower-bit quantization reduces memory, unsupported kernels or quality loss may worsen p95·errors. Evaluate capacity, execution path, and quality in the same candidate row, with independent thresholds.

Cost includes not only hardware prices but also power, storage, network, operations staff, and downtime. Do not let a candidate that fails required quality, p95, memory, or availability survive on the overall score because it is cheap. Compare cost per request, headroom at the target load, and maintenance difficulty only among passing candidates. Prices and cloud rates change by date and region, so do not embed permanent rankings in the official catalog text; manage them as separate, dated outputs.

Actually run the canary and rollback before promotion. Keep the previous artifact, runtime, and config in an accessible registry, and send the same inputs that failed on the new candidate to the previous backend to confirm recovery. Record rollback time, cache and index compatibility, and the responsible owner in the runbook. After full cutover, keep monitoring quality, p95, errors, and drift; if a threshold is exceeded, do not automatically continue risky actions, and instead limit traffic or revert to the previous candidate.

Why does this happen?

Because a usable model must not only answer correctly but also respond stably under the target load and recover within a set time after a failure.

When is it a problem?

Looking only at one warm tokens/s result and file size leaves cold-start, p95 queue, maximum-context OOM and long-running failures undiscovered until deployment.

Common beginner misconceptions

The top-ranked candidate for quality does not automatically run fastest or at the lowest operating cost on the target hardware.

How to verify it yourself

Pass quality, p95, peak and error gates under repeated load with the same manifest, then measure recovery time for the same failing input using the previous artifact.

The catalog schema holds the role, official source and revision, artifact provenance, license conclusion, runtime stack, evaluation set version, quality, p95, and memory results, and owner. Divide status into evidence-based stages such as exploring, technical review, license hold, evaluation hold, canary, approved, and retired. Do not read blanks as passes; record evidence gaps that state which model card, license, runtime, or evaluation is missing. A single screen should show the current recommendation together with the conditions under which it is valid.

Each row needs a last review date and re-review triggers. Triggers include changes to the official repository·license, a new model revision, updates to conversion tools·runtime·driver, hardware replacement, prompt·retrieval changes, and drift in task data. Rather than failing a row automatically just because a date has passed, set the review interval according to how likely things are to change and the task risk. For high-impact events such as a security advisory or a license change, you must be able to put the related artifacts on hold immediately without waiting for the regular review.

When a new release comes out, do not overwrite the previous row. Create a new row or version for the new source revision and artifact hash, run the same gold set and load, and record the diff. Do not delete failed results or reasons for retirement either, so that the same candidate is not reviewed repeatedly and past decisions are not lost. When the evaluation set changes, do not rank old and new scores together directly; mark the dataset version and rubric change.

Final approval requires separated responsibilities. Technical reviewers assess provenance, runtime and measurements; task owners assess gold answers and failure costs; security and legal reviewers assess files, code and terms. Preserve an independent review check even if the approver also conducted the experiment, and do not treat browser checkboxes or screenshots alone as actual evidence. A catalog becomes an operational asset when another person can reproduce the conclusion from sources, manifests, raw results and rollback logs.

Why does this happen?

Because models, licenses, runtimes, and task data keep changing, recommendations go stale quickly without approval conditions, owners, and review dates.

When is it a problem?

Overwriting existing rows with the latest results makes it impossible to know which artifact was actually deployed and why it was retired, so regressions and audits cannot be handled.

Common beginner misconceptions

A model that passed once is not permanently approved for every new revision, runtime, and changed task.

How to verify it yourself

Pick one row and check whether a third party can follow its owner, source and artifact hashes, license, raw evaluation, approval conditions, triggers, and rollback evidence.

Users feel they chose a single model name, but several layers shape the actual response. The weights, config, tokenizer, and processor downloaded from a repository form the artifact layer, and the chat template turns roles and special tokens into model input. The runtime loads these, batches requests with a scheduler, and runs backend kernels on a device such as a CPU or GPU. In server mode, an API parser, streaming, queuing, and metrics are added. The same model weights can differ in quality, memory, and latency depending on which layer changed.

Ranking Ollama, llama.cpp, MLX LM, Transformers, and vLLM in a single “fastest first” order loses these layers. Their design scopes differ: Ollama centers on model management and the local API experience, llama.cpp on direct control of GGUF and diverse backends, MLX LM on the Apple silicon path, Transformers on broad model·training control in Python·PyTorch, and vLLM on online serving. A convenient choice for one purpose does not necessarily have the features essential for another.

Isolate failures by layer evidence. For mixed roles, inspect rendered chat templates; for load failures, architecture/files and runtime support; for slow responses, queues, device placement, kernels and transfers; for format omissions, API parameters, sampling and model outputs. Saying only 'the same model answers differently' cannot identify changes in artifacts, templates or scheduling. Link input to response with request IDs and manifests.

Runtime changes are software deployments. Manage binary·container digests, package·driver versions and build flags, model·artifact hashes, config, and API contracts as versioned artifacts. Do not assume notebook latest packages or desktop auto-updates match production. Preserve previous images·configs·artifacts and test the same failing input on the previous stack to explain improvements and regressions.

Why does this happen?

Because separating the layers that produce a response is what allows model capability problems to be correctly distinguished from template, runtime, backend, and API problems.

When is it a problem?

Recording only the model name prevents reproducing which artifact·configuration·device path caused quality·latency regressions after a runtime update.

Common beginner misconceptions

Using the same weight file does not automatically make prompts, memory usage·performance, and API results the same across all runtimes.

How to verify it yourself

For one current request, record the artifact, tokenizer and template, runtime and scheduler, backend and device, and API response in order, with versions and logs.

Personal terminal chat emphasizes installation, model replacement, and a single request’s TTFT·decode and privacy. Research·training needs Python model objects, processors, gradients·checkpoints, and custom code. Desktop applications emphasize localhost API startup·streaming and idle memory; team services require authentication, queue·rate limits, p95·availability, and rolling updates. Calling all four workloads “running a local LLM” blends each candidate’s strengths and defects into an average.

The manifest contains the exact model·revision·artifact hash, tokenizer·template, input·output tokens, batch·concurrency, sampling, and hardware. Also pin normal·boundary·failure inputs, including short chats, long documents, missing evidence, cancellation, invalid schemas, and concurrent mixes of long and short requests. A personal demo runs at concurrency 1 with short prompts, while production traffic accumulates queues and KV cache, so the two cannot be compared with the same single token/s measurement.

Describe required functionality through client behavior, not endpoint names. Specify real requests and expected responses for streaming chunk order, usage fields, structured-output schemas, tool-call arguments, embedding dimensions, image inputs and error statuses. “OpenAI-compatible” is a useful starting point, but supported endpoints and parameters can differ by runtime and version. Automatically test the contracts clients depend on, and explicitly hold unsupported features.

Set non-functional requirements in advance as well. Fix minimum quality and format, p95 TTFT and total latency, throughput, peak memory, error rate, and recovery time before seeing candidate results. Document LAN and internet exposure, authentication and prompt log privacy, offline requirements, and model download paths. Do not approve a runtime that fails a required condition on its overall score just because it is easy to install or has a high average speed.

Why does this happen?

Because runtimes have different design goals, fix your own workload and client contract so that comparison results apply to real use.

When is it a problem?

Approving team service from a one-user demo leaves queue, p95, authentication, cancellation and overload issues undiscovered until deployment.

Common beginner misconceptions

Similar APIs or a successful localhost response do not establish that all production feature, security, and availability contracts are met.

How to verify it yourself

Build a table of user and request distributions, exact features, and quality, p95, memory, security, and recovery thresholds, and approve it before looking at candidates.

Runtimes do not read every model file interchangeably. llama.cpp centers on GGUF, MLX LM uses MLX-compatible artifacts, and Transformers and vLLM follow supported Hugging Face model and quant paths. An Ollama package also contains the actual base artifact plus template and parameters. Even within the same family, differences in precision, tensor layout and metadata, or processor files change the load path, kernels, and quality. In the candidate table, record the source revision and actual bytes and hash, not just file extensions.

Conversion creates a new artifact. When converting original safetensors to GGUF, MLX or runtime-specific quantization, link converter/quantizer versions, commands, options, calibration and source/output hashes. Check that community conversions have clear sources and recipes. A smaller file loading successfully does not establish equivalent quality or supported features; rerun the same gold set and runtime-log checks as the original baseline.

Verify that the tokenizer, chat template, and processor match the weights. Changes to instruct-model roles, BOS/EOS and stop behavior, vision image processors, or embedding pooling and normalization change model inputs and outputs even for identical displayed text. Do not assume the runtime automatically selects the repository template. Save tokens and rendered prompts immediately before model calls. If you override a template, record its source, reason, and regression results in the manifest.

Custom code and file security are part of runtime selection. Transformers remote code, pickle-family artifacts, and installation scripts broaden the execution boundary. Review required code, use isolated environments and least privilege, and retain only allowed files in the offline registry. Check whether the Runtime automatically updates or downloads from remote repositories; in production, use immutable revisions, controlled caches, and network policy to prevent unexpected artifact changes.

Why does this happen?

If the artifact and input-processing components differ by runtime, the actual computation graph, prompt, and quality differ even within the same family.

When is it a problem?

Pinning only the weights while mixing tokenizers, templates, and processors can silently change role handling, termination, vision, and embedding results.

Common beginner misconceptions

Matching model names and quant labels do not make GGUF·MLX·Transformers artifacts bit-identical or compatible with every runtime.

How to verify it yourself

Record source-to-converted-artifact hashes and tokenizer, template, and processor revisions. Compare rendered inputs immediately before runtime against the baseline.

Ollama's official API provides model interactions such as generate·chat·embed at the default localhost address after installation. It makes personal experiments and desktop app connections quick to set up, but simplicity does not remove run conditions. Preserve the model name·tag, digest and details, Ollama release, the Modelfile's FROM·TEMPLATE·PARAMETER·SYSTEM, and the API request. If you record only the `latest` tag, a later pull may fetch a different artifact and the result may not be reproducible.

Distinguish the model architecture's maximum context from the current runtime allocation. Official Ollama documentation notes that a larger context increases memory requirements and recommends checking PROCESSOR and CONTEXT in `ollama ps` for allocated length and offloading. A short prompt running successfully does not prove target context and concurrency stay on GPU. Record system RAM/VRAM peaks, processor split, TTFT, and decode at the longest input.

Make local and cloud data paths explicit. Even with the same API shape, a localhost model and a remote cloud model have different boundaries that input crosses. Confirm the actual endpoint through app settings, the model URI, and network logs, and align it with sensitive-data policy. Changing the listen address for LAN use changes the localhost-only threat model, so review an authentication proxy, TLS, rate limits, and per-client access as a new deployment.

Approve an Ollama candidate based on the actual client contract and failure recovery. Test the required features among streaming, structured output·tools·Vision·embeddings, as well as behavior for invalid requests·timeouts and during model load. After an update, run regression on the same gold set and p95 to see whether template·context·backend changed. Confirm that reverting to the previous Ollama version·model digest·Modelfile recovers the same failing input before deciding the desktop or team usage scope.

Why does this happen?

Beneath a convenient abstraction, the artifact·template·context·offload and endpoint determine the result, so the actual state must be recorded.

When is it a problem?

If you record only the tag and "response succeeded," you cannot find why quality or p95 changed after an update switched the model, context, or CPU offload.

Common beginner misconceptions

The fact that Ollama runs easily on localhost doesn't automatically guarantee 100% GPU, maximum context·production security, and original license.

How to verify it yourself

Store exact digests, Modelfile, API requests, and PROCESSOR/CONTEXT from `ollama ps`, then repeatedly measure quality, peak usage, and p95 for the longest workload.

The official llama.cpp repository provides GGUF, CPU, Metal, CUDA/HIP, Vulkan, other backends, and CPU+GPU hybrid inference. This list does not guarantee equal support for every model architecture, quantization, operator, and endpoint on every backend. Pin the repository commit, compiler and build flags, binary and container digests, backend, and runtime logs. Even for prebuilt binaries, check the release asset, hash, and CPU instruction set.

Check architecture, quantization, tokenizer and template in GGUF metadata, and record model and mmproj file hashes. If converted directly from the source model, preserve the recipe. With convenience downloads such as `-hf`, check the exact repository, quantization selection and cache object instead of depending on changing defaults. A llama.cpp-compatible community artifact is not necessarily the official model producer's original; license and provenance are separate gates.

Offload is not just a layer count. Use startup logs to check tensor·operator placement on CPU·GPU, host-device transfers, system RAM·VRAM peaks, and changes across context·batch sizes. A hybrid setup may run a model larger than VRAM, but link and CPU bandwidth can push TTFT·decode·p95 beyond targets. Compare full-GPU, hybrid, and CPU baselines on the same workload, and do not hide unexpected fallback.

When using llama-server, separately verify operational features such as the API, parallel decoding, continuous batching, and metrics. Check the listen address, authentication and request limits, context sharing, slot and queue behavior, and health and readiness. Do not carry a single-user CLI success over as approval for a multi-user server; measure p95, errors, and cancellation at each concurrency step. Before updating, confirm you can roll back to the previous binary, preset, and GGUF and recover the same failing request.

Why does this happen?

Because llama.cpp offers so much freedom in build, backend, offload, and options that support and performance results cannot be reproduced unless the exact combination is recorded.

When is it a problem?

Looking only at GGUF loading and GPU activity misses p95 bottlenecks caused by CPU fallback for some operators, hybrid transfers and server queues.

Common beginner misconceptions

A supported-backend list does not mean every model, quantization format and feature is fully accelerated and production-ready on that device.

How to verify it yourself

Store commit, build flags, binary hash, GGUF metadata/hash, and startup placement, then repeatedly measure CLI and server workloads separately.

The official MLX LM repository provides a package for text generation, quantization, and LoRA·QLoRA fine-tuning with MLX on Apple silicon. Do not apply NVIDIA CUDA commands or kernels merely by renaming them. Record the exact Mac chip·memory·OS, Python·MLX·MLX LM versions, source model, and MLX artifact hash. For community conversions, check source revision·quant recipe and tokenizer·template, and compare quality against the original baseline.

Unified memory makes access to a large pool convenient, but the OS, applications, weights, KV cache, activations, and workspace share the same capacity and bandwidth. Measure per-phase peaks for generation, batching, and fine-tuning, and check memory pressure, swap, Metal execution, and thermals. Do not extrapolate one successful prompt generation to large batches, long contexts, or completed LoRA jobs. When comparing with llama.cpp Metal on the same Mac, control for artifact and prompt differences.

Separate the purposes of MLX LM commands. Chat REPL centers on session context and personal interaction; generate on reproducible prompt execution; batch on throughput; and fine-tune on dataset·checkpoint·validation. Pin sampling·seed·template, batch·sequence, and checkpoint recipes. For training candidates, separately test job time, activation peak·disk, resume, and adapter rollback as well as inference quality.

The official MLX LM server documentation states that it implements only basic security checks and is not recommended for production. The localhost API is convenient for development, but do not expose it directly to a LAN or the internet. If production is required, choose a separate supported path that provides authentication, TLS, request limits, and observability, or validate a proxy and threat model. Do not treat success on an “OpenAI-like” endpoint as approval for availability or security.

Why does this happen?

Because MLX LM's advantages are tied to Apple silicon and the MLX workflow, and generation, training, and the server have different operating conditions.

When is it a problem?

Approving batch·LoRA and external APIs from a single generation run on a Mac means memory pressure, job failures, and missing authentication are discovered only after deployment.

Common beginner misconceptions

Sufficient unified memory or a running MLX LM server does not automatically provide every CUDA workflow or production security.

How to verify it yourself

On the exact Mac, MLX artifact, and version, measure generation, batch, and training separately, and for servers check the official limits and actual network and security boundaries.

Transformers flexibly handles Hugging Face models, tokenizers, and processors in Python and PyTorch and connects them to generation, evaluation, and training code. Do not interpret this flexibility as automatic optimization on every device. Align supported Python and PyTorch requirements from the installation documentation, GPU drivers, and device checks with exact versions. Preserve virtual environments, lockfiles, and package hashes, and keep source-install results separate from stable-release results.

`from_pretrained` links a repository to the local cache. In production, pin the exact revision, allowlisted files, and local cache objects, and check offline mode·network policy. Make sure an update to the main branch does not automatically change the deployment artifact. Match the tokenizer·processor and config revisions to the weights, and put `trust_remote_code`, pickle, and custom extensions through source review·sandboxing and least privilege.

Using Transformers as a baseline allows detailed observation of the original precision, architecture, and model outputs. Save rendered templates, token IDs, dtypes, device maps, layer placement, and generation configurations. If GGUF, MLX, or vLLM candidate results differ, compare them with the Transformers baseline using the same source and prompt to isolate artifact quantization, template, and scheduler effects. The baseline itself also varies with drivers, kernels, and sampling, so pin its versions.

A Python notebook or pipeline is not a production service. When a multi-user API is needed, choose a serving layer that provides queues, async cancellation, authentication·rate limits, health·metrics, and process isolation. Do not bind research code directly to a web port. Separate training·evaluation from the serving environment while sharing the model·tokenizer·gold set contract, and automatically check for regressions that updates cause in the downstream runtime.

Why does this happen?

Because Transformers offers broad model control, the environment, remote artifacts, and device mapping directly affect results and security.

When is it a problem?

Running unpinned packages, main revisions, and remote code directly in notebooks can leave you unable to reproduce results and cause you to miss supply-chain risks and device fallback.

Common beginner misconceptions

A Transformers pipeline running once does not mean the script is a multi-user production server or the most optimized runtime.

How to verify it yourself

Save the environment lock, exact model and processor revisions, device map, and rendered prompt, then rerun the baseline in an offline clean environment.

The official vLLM documentation broadly covers GPU, CPU, and TPU installation, supported models, and online serving, pooling, quantization, and observability features. However, do not assume that features in the latest documentation apply retroactively to the older version you deployed. Check the container digest, vLLM, PyTorch, and driver versions, GPU and topology, model and quantization artifacts, and attention backend against the exact support page. Distinguish the risks and rollback of nightly and source builds from those of stable images.

OpenAI-compatible servers help client migration, but verify the actual endpoint contract. Pin the requests the app uses among chat·responses, streaming chunks, structured output, tool-call·reasoning parsers, embeddings, and usage·error fields. Model-specific chat templates and parser support may be needed, and options may vary by version. Test with invalid·boundary requests whether unexpected fields are silently ignored.

Measure serving performance while the scheduler batches multiple sequences. At concurrency 1 and representative, target, and overload levels, use realistic input and output length distributions to measure request throughput, queue time, TTFT, inter-token latency, p50, p95, p99, peak memory, and errors. Even if continuous batching increases total tokens/s, long requests may worsen the p95 for short requests, or KV-cache pressure may block admission. Also verify resource reclamation after cancellations and timeouts.

Production needs authentication, rate limits·quotas, readiness during load, metric·log privacy, and rolling restarts. Test server statuses and retry safety during OOM·worker crashes and model-load failures. Distributed execution adds topology·communication and node failures as variables. Observe quality·schema·p95·errors in canary and measure the time to return traffic to previous images·artifacts.

Why does this happen?

vLLM's value shows in concurrent request scheduling and serving features, so only reproducing actual clients, load, and operational failures provides grounds for selecting it.

When is it a problem?

Looking only at single-prompt token/s misses queue and tail latency, unsupported parsers and endpoints, and worker and request recovery after OOM.

Common beginner misconceptions

“OpenAI-compatible” and “high-throughput” do not automatically guarantee support for every model·quant·client parameter or production security.

How to verify it yourself

On the exact image and support matrix, run client contract tests, measure quality, p95, peak, and errors at each concurrency step, and run canary and rollback.

For a fair comparison, fix the source model and artifact, tokenizer and template, prompt, output, and sampling settings, context, batch, and concurrency, and the hardware. If runtimes support different artifacts and cannot use the same quantization, disclose the source revision, conversion recipe, and byte and quality differences, and keep a common Transformers or existing runtime baseline. Do not show only the candidate's best-performing settings; record the common settings and tuning differences together. Repeat at least 3 times to separate cold and warm runs from variance.

The measurement table includes download, loading, compilation, warm TTFT, prefill, decode, end-to-end and queue time, and request and token throughput. Link peak VRAM and RAM, CPU/GPU placement and transfers, utilization, power, temperature, errors and cancellation. Examine p50, p95, p99 and distributions by input length and concurrency rather than one average. Approve performance results only when the same candidate’s task quality, evidence, schema compliance and safe refusals pass predefined thresholds.

On failure, return to the smallest passing state. Verify exact outputs and device paths with short text, concurrency 1 and default precision, then add context, batch/concurrency, vision, tools and schemas individually. Find the first evidence that changed with symptoms among startup, rendered prompts, operators, memory, queues and networks. Hold combinations that succeed accidentally after changing model, quantization, runtime and driver together because they lack root-cause evidence and a reproducible recipe.

Before promotion, restore the previous runtime image and binary, config and API contract, and artifact. Send the new candidate's failing inputs to the previous backend and test whether quality, format, and p95 recover and whether the traffic switch and cached and in-flight requests are handled safely. Switch over fully only after the canary, and set regression triggers for version, driver, model, and task changes. The final conclusion is not a brand ranking but a pass or hold, with limitations, for the exact workload, stack, and date.

Why does this happen?

Because performance gains become operational value only when runtime differences are separated from artifact and configuration differences and you can recover from real incidents.

When is it a problem?

Changing several variables at once and keeping only the successful values makes it impossible to find the cause of regressions or to return safely to the previous service.

Common beginner misconceptions

One best tokens/s result or successful model load does not represent quality, p95, memory, errors or production readiness.

How to verify it yourself

Repeat normal, boundary and failure cases under the same manifest, and reproduce small-baseline recovery, canary testing and rollback to the previous stack using the same inputs.

For a personal API bound only to localhost, processes on the same host are the main clients, but exposing it to a LAN or the internet brings in untrusted clients and networks. Changing only the listen address and firewall is not a feature change but an expansion of the security boundary. Draw a data flow showing who can use which models, documents, and tools, and where inputs and outputs pass through and are stored. Confirm the default bind, authentication, and proxy configuration of Ollama, llama-server, MLX server, and vLLM in the exact release documentation and on the actual socket.

Distinguish authentication from authorization. Even with a valid API key, the server must check whether the user has the right to access the specific model, adapter, RAG documents, and tools. Do not put a master key in a browser client; use short-lived, least-privilege credentials and TLS. Hiding model names or making URLs hard to guess is not access control. Place admin endpoints, model pull and delete, and file tools on a narrower network and role than general inference routes.

Limit request resources. Without limits on input context, output tokens, image size·count, batch·concurrency, queue·timeout, and per-user quotas, a few long requests can exhaust KV cache·memory. Test that slots·memory are reclaimed after cancellation, and that overload returns a clear status without triggering infinite retries. Rate limits should reflect not only average request counts but also token·memory cost and high-priority traffic.

Logs and metrics can contain prompts, documents, tool arguments, model and adapter identifiers, and user identities. Minimize collection of raw content, redact secrets and personal information, and define access, retention, and incident procedures. Do not leave debug mode permanently on in production; trace samples must also follow data policy. Even after passing the threat model, authentication proxy, and limits, manage the runtime's own vulnerabilities, dependency updates, and rollback separately.

Why does this happen?

Because exposing a service on the network changes who can access it, the attack surface, and resource contention, the success of a personal run cannot establish safety.

When is it a problem?

Changing only the bind address can expose unauthenticated model and administration endpoints while introducing OOM from long requests and leakage through prompt and document logs.

Common beginner misconceptions

Using a local model does not mean that data is safe or that permissions are automatically isolated per user when the API is exposed to the LAN.

How to verify it yourself

Test the actual listening sockets, proxy and firewall, per-user permissions, token and queue limits, and redacted logs using an approved account, an unapproved account, and an overload request.

Registry rows hold purpose, owner, runtime·binary/container digest, model artifact·template, hardware·driver, client contract, and quality·p95·memory thresholds. They also link development·canary·production environments and traffic ratios, the latest evaluation, and the rollback target. A single line such as “currently using vLLM” does not show which version·model and settings actually respond to users. Deployment inventory and metric labels must use the same immutable IDs to narrow the impact during an incident.

Observability must explain the user's request path. Link request rate and queue depth; TTFT, inter-token, and end-to-end p50, p95, and p99; token throughput; errors and cancellations; and VRAM, RAM, and device placement. Measure model quality and schema failures with samples separate from infrastructure metrics while protecting privacy. Align alert thresholds with predefined service levels and error budgets rather than simple CPU utilization, and link rising queues and OOM to runbooks for concurrency limits, smaller models, and rollback.

Cost includes not only runtime license prices but also GPU and CPU time, idle and load memory, power, storage, network, and operations staff. For a single user, setup and update time matters; for a team service, cost per request, spares, and availability matter. Compare costs only among candidates that pass required quality, p95, and security, and do not approve a failing candidate because it is cheap. Attach the measurement date, traffic, and hardware, and recalculate after workload changes.

An upgrade is a new candidate, not an overwrite of an existing record. When releases, drivers, models, templates, or feature flags change, check official support and vulnerability notices, then run the same regression, load, and failure tests in a canary. Before retirement, check client and artifact dependencies, caches, adapters, and rollback retention. Preserve evidence after traffic reaches zero and access is blocked. Do not delete the previous runtime so early that a failed canary has no rollback path.

Why does this happen?

Because runtime and deployment conditions keep changing, approval results go stale quickly without the current stack, thresholds, owner, and re-review triggers.

When is it a problem?

Overwriting existing rows with the latest update and deleting the previous image makes it impossible to trace the scope and cause of quality or p95 regressions, or to roll back.

Common beginner misconceptions

A runtime that passed in production once is not permanently approved for every new version, driver, model, and traffic distribution.

How to verify it yourself

Trace the immutable runtime·model stack and metric·owner·rollback from a production request ID, and rehearse update·retirement procedures with a canary failure.

Training may seem to produce one weight file, but production runs a combination of artifacts. A PEFT adapter lacks base weights and requires its adapter configuration and exact base model. A merged model contains full weights combining the base and update, while GGUF contains architecture and tokenizer metadata plus inference tensors. An Ollama package reassembles FROM sources, ADAPTER, TEMPLATE, SYSTEM and PARAMETER. Runtime, storage, licensing and rollback depend on which node you deploy.

The graph starts with exact base weights, configuration, tokenizer, chat template, adapter and dataset/training run. A merge edge converts base plus adapter to merged safetensors; a converter edge produces high-precision GGUF; and a quantizer produces Q4/Q5 candidates. A package edge combines the chosen source and runtime configuration into a manifest/blob. Record each edge's tool commit, dependencies, parameters, environment, input/output SHA-256 hashes, raw logs and evaluations.

Without this graph, when model-final.gguf fails, you cannot tell which base, adapter, template, or quant recipe caused it. Even if a community file's name cites the same source model, its actual revision and conversion may differ. An approved registry keeps node digests and parent digests immutable and moves only the dev, staging, and production aliases to reviewed digests. Never overwrite bytes under the same name.

The artifact owner defines each node's intended use, retention, access, and re-review triggers. Connect the base license and acceptable use, fine-tuning data rights, merge, quantization, and redistribution conditions, and model card limitations. A successful conversion run or included LICENSE text does not grant actual usage rights. Keep quality, security, and license decisions and build provenance as independent evidence.

Why does this happen?

Because deployed behavior combines the weights with the base, tokenizer/template, quantization, and runtime settings, and recovering from errors requires tracing those dependencies back.

When is it a problem?

Keeping only the final file loses source, tool, license and previous-combination information, making reproduction and incident response impossible.

Common beginner misconceptions

Adapters, merged safetensors, GGUF, and Ollama packages are not identical bytes and behavior with merely different extensions.

How to verify it yourself

Starting from the production digest, confirm that following parents reaches the exact base·adapter·template·converter·quant·package and each evaluation report.

The Hub's official download guide allows branch, tag or full commit-hash revisions instead of the default latest main; commit hashes must be full length. Pin release inputs using repository ID/type, the full commit and an allowed file list. Treat the returned version-aware cache path as read-only input and do not modify cache files directly. Record the snapshot's configuration, tokenizer, weights, model card and code-file inventory.

SHA-256 checks whether file bytes match the bytes identified by the manifest. Calculate it before and after download, during registry upload and node pull, and immediately before loading to detect partial transfers, corruption, and tampering. However, a hash alone cannot establish trust if an attacker changes both the file and its manifest hash. Approved channels, signature and attestation verification, and reviewer policy provide additional grounds for trusting the source manifest.

SLSA provenance describes how an artifact was produced so consumers can verify that it is the expected build and rebuild it if needed. Apply the build type, external parameters, and resolved dependencies from buildDefinition, and the builder and invocation from runDetails, to model-conversion records. Link the converter command, base and adapter digests, quantization type, imatrix, and output subject digest so reviewers can identify hidden inputs.

Provenance does not mean the build is safe or the model is accurate. Verify builder identity·isolation, parameter completeness, and attestation signature policy, and disclose the reproducibility limits of local manual runs. Quality evaluation·license review·malware scanning are separate gates. The registry UI shows digest·parent·builder·review status and evidence on one screen, but does not merge all of their meanings into a single green badge.

Why does this happen?

Artifacts with the same name can change, and byte identity alone does not reveal how they were created or who approved them.

When is it a problem?

Deploying only the latest tag and a filename means a restart can fetch different bytes, and a hash alone cannot prove that it came from an approved build.

Common beginner misconceptions

A matching SHA-256 digest means that the file matches the expected bytes; it does not mean that the file is safe, accurate, or legal.

How to verify it yourself

Check the full source commit, file digest, the inputs, builder and parameters of signed/approved provenance, and independent quality and license decisions separately.

Restoring a Python object graph with Pickle can trigger imports and callable execution, creating arbitrary code execution risk. The Hub pickle security documentation warns that its scanner extracts imports and flags suspicious items but is not 100% foolproof. Do not approve unknown files that may be pickle-based, such as .bin·.pt·.pth·.pkl, as production identities; perform any essential legacy conversion only in an isolated sandbox without network·credentials, under a disposable user.

The official Safetensors documentation describes a simple format for safely storing tensors compared with pickle. It avoids deserializing code objects, but tensor values may contain NaNs or have unexpected shapes, and file sizes can exhaust resources. Before loading, inspect header size, metadata, the key allowlist, dtype, shape, tensor count, and total bytes. Pin the parser and framework versions and set device-memory limits.

Identify formats using actual magic/header bytes and a parser, rather than the extension. A safetensors directory may also contain configuration, tokenizer files, custom modeling code and pickle files, so inspect the repository inventory. Deny trust_remote_code and executable scripts by default, and verify architecture support in official code and versions. Block unexpected files, symbolic links, path traversal and archive extraction in quarantine.

Behavior evaluation remains necessary after safe loading. Tensor formats cannot prevent weight backdoors, data leakage or malicious tokenizers. Check golden inputs, critical/safety/resource sets, provenance and licenses. Record transitions from quarantine to verified input, candidate registry and approved release; automatically hold on format-scanner failures or unreviewed dependencies.

Why does this happen?

Because loading weight files uses production process permissions and large amounts of memory, and unsafe deserialization, resource abuse, and behavioral risk sit at different layers.

When is it a problem?

Trusting only an antivirus or Hub scanner green check to load a pickle, or skipping companion code, shape, and behavior checks because a file is named safetensors, leaves security and quality defects.

Common beginner misconceptions

Safetensors is not a trusted-model certificate; it is a tensor data format designed to reduce pickle's code execution surface.

How to verify it yourself

Check default denial of unknown pickles, safetensors header/key/shape/dtype/size inspection, the repository inventory, and sandboxed fresh-load and behavior evaluation.

The official PEFT checkpoint documentation describes adapter_model.safetensors, adapter_config.json, and the README model card, and states that the adapter state_dict contains no base parameters. Compare base_model_name_or_path and revision and peft_type·target_modules·rank·alpha·modules_to_save in the adapter config against the training run manifest. Supplement a mutable base name or null revision with an exact weight hash, and confirm that checkpoint keys·shapes match the actual target report.

Tokenizers, chat templates, and special tokens are dependencies of adapter behavior. If training resized the vocabulary or trained new embeddings or lm_head, include tokenizer files and modules_to_save weights. Compare golden token IDs produced by rendering raw messages with the same template and serving outputs. Copying only the adapter and changing the tokenizer to the latest base version can break role and target behavior even if loading succeeds.

In a fresh process, load the exact base from an immutable cache/registry and inspect adapter keys, missing/unexpected keys and dtypes. Compare golden logits and generations for base-only, adapter-enabled, adapter-disabled and training-reference runs. With a multiple-adapter runtime, recognize that an adapter name is not checkpoint identity and log the resolved adapter digest for every request. Also test hot swaps and restarts for stale adapters remaining in worker caches.

Ollama’s import documentation also states that ADAPTER’s FROM must match the base used in fine-tuning or results may be erratic. Check supported architectures·adapter formats and quant compatibility for the exact Ollama version. A QLoRA source does not imply support for every runtime import. Approve high-precision merge or separate conversion candidates through official paths and evaluation.

Why does this happen?

Because the tensor positions and learned behavior of an adapter update depend on the exact base structure and token sequence, and the file alone contains no base.

When is it a problem?

Attaching it to a similarly named base or latest tokenizer causes shape errors or silent quality drift, and the original training state cannot be recovered.

Common beginner misconceptions

An adapter_config that names a base, or a successful load, does not guarantee exact revision and template compatibility or identical quality.

How to verify it yourself

Compare the training base, tokenizer/template digest, adapter config/key, fresh base-only, enabled, and disabled outputs, and the runtime resolved digest.

The PEFT checkpoint documentation explains that merge_and_unload can save a full model, but PEFT-specific methods and unmerge·multiple-adapter·disable features are lost, and not every method·quant setting supports merging. Merge inputs are the exact high-precision base and adapter digest; record the device·dtype·merge tool commit. Do not force a merge into an already quantized base, and do not approve a community-merged file without a parent as a source.

Freeze the dynamic-adapter reference before merging. Preserve the same base·adapter, tokenizer/template·decoding, and raw outputs·logits for frozen task·critical·general sets. Inspect merged tensor keys·shapes·dtypes, missing/unexpected entries, NaN/Inf, and file-shard indexes, then calculate SHA-256. Load it as a full model in a fresh process and evaluate allowed logit/output differences from the reference.

A merged model includes all base weights, so storage·access·redistribution differ from distributing only an adapter. Review the base license, adapter data·license, and derivative-distribution terms against actual use·region·commercial plans. Record parent digests, merge recipe, evaluation·limitations, and dynamic-adapter source in the model card. README metadata does not replace legal approval.

Preserve the original base and adapter bundle for rollback, and do not overwrite the merge output. If a merge bug is found, take down the candidate alias, recover to the dynamic previous path, and retest the exact failing input. Set a policy so the next GGUF converter accepts only the approved merged digest, which keeps a rejected merge from spreading to downstream quants and packages.

Why does this happen?

Because merge changes parameter bytes, runtime features, and distribution rights, and becomes the source for every later conversion, it needs its own release boundary.

When is it a problem?

Deleting the source after seeing only a successful merge makes it impossible to compare against the dynamic reference or to unmerge·rebuild when quality drift·license issues arise.

Common beginner misconceptions

Merge is neither compression of an adapter file nor a metadata change that is fully reversible for every PEFT method.

How to verify it yourself

Compare tensors·logits·task·critical·license between the pinned base+adapter reference and the merged fresh load, and test rollback to the original bundle.

The GGUF specification defines a magic/version header, key-value metadata, tensor information and aligned tensor data. Standardized keys include general.architecture, quantization_version, alignment and model name, version and basename. Use official information tools/parsers to check the header, tensor count, offsets, types and required architecture keys. Do not reject every unknown key, but place missing required values, unsupported versions and invalid offsets on hold.

Use a pinned llama.cpp converter commit and Python dependencies to convert approved safetensors/config/tokenizer digests into high-precision GGUF. Record commands·environment·stdout/stderr, shard·special-vocabulary handling, and output hashes in build provenance. Do not ignore custom-architecture warnings, missing pretokenizers·chat templates, or tensor-mapping mismatches. Do not directly edit source files or manually hex-patch converter output.

Use BF16/F16 GGUF as the conversion reference where possible. Compare the source Transformers/PEFT merged model and high-precision GGUF using identical rendered tokens and decoding to verify vocabulary, rope, architecture mapping, logits, and frozen task/critical outputs. If this stage fails, fix converter, metadata, or source causes rather than changing quant type. Only a digest that passed conversion becomes quantizer input.

Even when a GGUF file contains template metadata, check which precedence and fallback the target runtime uses. Pin the runtime command, context, sampling, and backend commit, and compare the actual prompt dump or token IDs with the reference. When companion files or architecture-specific metadata are required, as for vision, embedding, or MoE, include them in the parent graph. Loading the file and succeeding on one prompt do not demonstrate full feature compatibility.

Why does this happen?

Because if the converter maps tensor, tokenizer, or architecture metadata incorrectly, behavior changes before quantization even starts, and the defect is copied into every later candidate.

When is it a problem?

Comparing only the Q4 result with the source cannot separate conversion mapping errors from quantization loss, which leads to tuning the wrong knob.

Common beginner misconceptions

A .gguf extension and a successful runtime load are not evidence that the source architecture·tokenizer/template·tensors were preserved exactly.

How to verify it yourself

First pass the high-precision GGUF's header, metadata, and tensor inventory checks, plus golden-token, logit, and task evaluations against the source.

The official llama.cpp quantize documentation explains that llama-quantize turns high-quality GGUF input into output such as Q4_K_M, and that quantization changes size and inference speed but can introduce accuracy loss. Record the tool commit, input digest, quantization type, threads and imatrix digest, include/exclude and tensor overrides, and output hash. Do not infer source precision, recipe, or tool version from the Q4_K_M string in a community filename alone.

Official guidance warns that requantizing already quantized tensors can yield substantially lower quality than direct conversion from 16/32-bit sources. Require explicit exception review if the parent graph does not reach a high-precision source or `--allow-requantize` was used. Generate Q4 and Q5 independently from approved F16/BF16 GGUF rather than deriving Q4 from an earlier Q5.

Split Evaluation into file parsing/model loading, golden tokens/templates, task/critical/general/long-context quality, and resource use. Compare raw outputs against the high-precision reference with identical prompts, decoding, and runtime. Set allowed quality drops, minimum schema/critical scores, and memory savings before results. On target hardware, repeatedly measure peak memory, first-token and prompt-processing performance, generation rate, p50/p95, and warm/cold states.

Choose an artifact that passes every must-pass gate and meets operational goals, rather than simply the smallest candidate. If Q4 reduces memory but fails critical JSON cases, while Q5 passes memory·p95 and quality requirements, Q5 is the approval candidate. When using Imatrix, record the representative source·rights·hash and leakage checks. Re-evaluate the same frozen set whenever the quantizer or runtime changes.

Why does this happen?

Because quantization is an irreversible precision change, and the source·recipe and per-tensor settings change quality·memory·kernel behavior.

When is it a problem?

Judging only by filename and size can mistake requantized artifacts or different imatrix/converter outputs for the same Q4 artifact and deploy critical regressions.

Common beginner misconceptions

Having the same Q4_K_M name does not mean every model·tool version produces the same bytes·quality·speed.

How to verify it yourself

On the target hardware, confirm the approved high-precision parent, the pinned quant command and output digest, and task, critical, memory, and p95 results relative to the reference.

The Ollama Modelfile reference defines FROM as the required base instruction, which can point to a supported safetensors directory or a GGUF file, and provides ADAPTER, TEMPLATE, SYSTEM, PARAMETER, and LICENSE. The import documentation warns that safetensors·GGUF adapters must use the same base used for fine-tuning as FROM. Check supported architectures and quant·adapter paths for each Ollama version before release.

TEMPLATE serializes roles and content into actual control tokens, and stop parameters change generation termination. Compare golden conversations from training and reference runs with Ollama prompts and token dumps. Adding SYSTEM or MESSAGE changes behavior, so evaluate it as a separate prompt candidate rather than convenience metadata. Pin defaults such as num_ctx, temperature, and top_k, and the request-override policy in the manifest.

The LICENSE instruction packages the model's license text; it does not replace legal or owner review. Connect base, adapter, dataset, merge and quantization licenses and the model card's intended/out-of-scope uses to the release decision. A public push is a state change requiring separate external-publication authorization; local package validation does not automatically authorize it.

After Create, export actual Modelfile, manifest, and blob digests and compare intended source digests, templates, and parameters. Repeat build, pull, and run on a clean host, logging resolved digests rather than package names or tags. Even if a registry alias points to the candidate digest, test restart and eviction with the same failing input to check whether workers still use cached old blobs.

Why does this happen?

Because an Ollama package combines weights with prompt serialization·generation defaults, final behavior can differ even when the upstream GGUF alone passes.

When is it a problem?

If you check only Create success and the package name, you discover a wrong base adapter, template or stop mismatches, and mutable tag drift in production.

Common beginner misconceptions

A single LICENSE or SYSTEM line in a Modelfile does not automatically establish actual redistribution rights·quality approval or identity with the training template.

How to verify it yourself

Check the resolved FROM/ADAPTER digest, golden TEMPLATE·stop·parameters, the license decision, and the clean build·actual worker digest.

Distinguish registry states such as quarantine, verified-input, candidate, approved, and revoked. A candidate record holds digest·size·format, parent graph, build provenance, file inventory·scanner, license decision, evaluation·limitations, owner·review date, and supported runtimes. Finishing an artifact upload does not automatically move the approved alias. Promote atomically only after every independent gate and approver has confirmed, and keep an audit log.

On an offline release set, compare base/dynamic adapter, merged weights, high-precision GGUF, quantized artifacts, and packages using the same template·decoding·runtime. Pass format·identity, task·critical·general, p95·memory, and fresh-load gates at every edge. Replay builds several times to explain expected deterministic digests or permitted metadata differences. Include corrupt downloads·digest mismatches and missing tokenizers as negative tests.

In a limited canary, check that the actual production endpoint uses the expected blob, template, and parameters, and observe schema and critical incidents, human corrections, p95, memory, and load errors. Apply predefined stop conditions instead of simply filling a quota of canary traffic or time. Register unexpected digests, license expiry, source revocation, runtime CVEs, model defects, and template changes as re-review triggers.

Rollback is not an alias edit: restore previous compatible base/adapter or merged·GGUF artifacts, tokenizer/template, Modelfile parameters, and runtime config·caches. Test candidate-failing inputs and normal regression sets several times within the recovery target. Inventory workers·caches·download mirrors for revoked digests, and record incident root causes, affected downstream nodes, and rebuild decisions.

Why does this happen?

Artifact defects spread to registries, caches, and workers, and replacing a single file may not restore compatible behavior.

When is it a problem?

Keeping only the previous GGUF while losing the template·runtime·adapter graph can leave the same error after rollback or introduce a new incompatibility.

Common beginner misconceptions

An immutable registry and a canary do not automatically make a release safe without quality and license review or an actual rollback drill.

How to verify it yourself

Repeatedly verify every gate in the approved record, the actual worker digest, and the failure set, normal set, and recovery time under a rollback to the previous full graph.