LLM principles and language

A Large Language Model (LLM) may look like a single program that understands written instructions, but a real service consists of several layers. When a user enters a question, the application assembles the system prompt (hidden operating instructions), the previous conversation, retrieved documents, and the question in a fixed order. A tokenizer, the component that splits text into pieces for the model, converts this text into an array of numbers called token IDs, and a runtime such as Ollama, llama.cpp, MLX, or vLLM performs the computation with the model weights. Finally, the reverse process turns the numeric pieces back into sentences people can read. Understanding a local LLM means looking at this entire flow, not just the model name.

The model weights at the bottom are a very large set of numbers adjusted during training. A file that stores only these numbers cannot take questions, draw an interface, or search the internet on its own. It needs a runtime to read the file format, a Central Processing Unit (CPU) or Graphics Processing Unit (GPU) to compute, a chat template to format inputs, and an app to show results. This is why the same model file can produce different speeds and answers when runtime settings, prompt formats, context lengths, or quantization methods differ.

Even when a user says “I want to put my data into AI,” first break down the function they want. Retrieval-Augmented Generation (RAG), which retrieves new facts and attaches them next to the question, fine-tuning, which adjusts answer format and behavior, and history management, which feeds earlier conversation into the next request, are different functions. Assuming the model has permanently learned a PDF you made searchable, or putting the latest internal policies into fine-tuning data to fix tone, makes maintenance difficult. Only with this distinction can you choose the right solution without overbuying tools and hardware.

Why does this happen?

Because the cause can be found only by separating which layer the problem arose in when an error occurs: the model itself, the runtime, the prompt, or the retrieved documents.

When is it a problem?

If you only record that “the AI is acting strange,” you may misdiagnose a wrong chat template or an outdated retrieved document as a model performance problem and end up buying more expensive hardware.

Common beginner misconceptions

Downloading a model file does not install an entire AI or train it on your data. It only prepares already trained numbers to run on your hardware.

How to verify it yourself

For the app you use, write down the model name, runtime version, context setting, system prompt, and whether retrieval is used. Anything you cannot fill in is a system layer that is currently invisible to you.

To a human reader, a paragraph of answer appears quickly, but inside the model small generation steps repeat continuously. First, the model reads the entire input and computes logits, the scores for the many token candidates that could come next. These scores are converted into a probability distribution, and one candidate is selected according to the configured selection rule. Appending the selected token to the existing text makes the input one piece longer, and the model computes the next candidates again with the extended context. The same computation continues until an end token is selected or the maximum output length is reached.

After 'It rained on my way home', plausible continuations might mention an umbrella, a taxi or wet clothes. The model does not look up a question's meaning in a dictionary and copy a finished answer; it estimates the next piece's probability from learned language patterns and current context. Lower temperature favors high-scoring candidates more consistently, while higher temperature increases the chance of lower-ranked candidates. This generation process naturally produces different wording for the same question.

You may wonder, “If it predicts the next token, how is it different from simple autocomplete?” The difference lies in the scale of training and the depth of internal representations. To predict the next token across vast amounts of text, code, tables, and explanations, the model must compress not only grammar but also relationships between entities, the purpose of the writing, facts that frequently appear together, and reasoning patterns into numerical relationships. As multiple Transformer layers combine these relationships, complex capabilities that look like translation, summarization, and question answering emerge. A simple training objective does not mean the model's internal computation or its results are simple.

Why does this happen?

Understanding token-by-token iteration lets you measure response speed in tokens per second and explain why longer outputs require more waiting.

When is it a problem?

If you mistakenly believe entire answers are stored, you cannot understand why temperature, context, and retrieved evidence change the results, and you become careless about fact-checking.

Common beginner misconceptions

Always choosing the most probable token does not make the whole sentence the most factual. Picking a natural candidate at each position is different from verifying external facts.

How to verify it yourself

In the course's token generation lab, change the selection rule while keeping the same sentence. You can see whether changing one earlier token changes the candidate distributions that follow in a chain.

Training calculates the difference between model predictions and training-data targets, then repeatedly adjusts weights to reduce that error. Large-scale pretraining requires many GPUs and substantial time, alongside data cleaning, distributed computation, checkpointing, and evaluation. Individual LoRA (Low-Rank Adaptation, fine-tuning a small set of additional weights) has a narrower scope but still requires data preparation, training, and evaluation procedures. Pasting a document into a chat window or downloading a model file does not change weights.

Inference is the process of loading already trained weights into memory and computing the answer to a new input. Running a model in Ollama to chat, or opening a GGUF file with llama.cpp, is inference. When a chat app saves earlier messages and prepends them to the next question, the model may seem to remember the past, but the conversation history has not become the model's permanent knowledge. If the history is not sent in a new conversation, that information is gone, and the model file remains unchanged.

RAG is useful for frequently changing facts such as current regulations or personal documents. Retrieve relevant document passages first, then send their source text with the question so the model answers within the evidence. Fine-tuning is more suitable for consistent response formats, tone, classification, and task procedures. A monthly price list may be managed with RAG, while consistently classifying customer inquiries into one of five company categories may be a fine-tuning candidate. Before choosing, ask whether the need concerns new facts, behavior, or temporary conversation memory.

Why does this happen?

Distinguishing the three functions lets you design the data update cycle, GPU requirements, personal data exposure points, and failure causes precisely.

When is it a problem?

Teaching current facts through fine-tuning requires retraining whenever information changes; trying to change tone through RAG can require repeating long instructions on every request.

Common beginner misconceptions

When people say they “trained AI on their data,” they have often just added a file search feature. Even if the results are useful, the two are not technically the same thing.

How to verify it yourself

Compare the model file's modification time and hash before and after a conversation, then ask about previous information in a new conversation. Without a separate training process, weights remain unchanged and information outside the supplied history does not persist.

Because the model is trained to predict the next token well, it is good at producing explanations that sound natural in context. But naturalness is no guarantee of factual accuracy. When asked about events after its training cutoff, exact version numbers, rare specialist knowledge, or papers and sources that may not exist, a model can fill the gaps with plausible patterns. It may accept a false premise in the question as given or mix up the specifications of similar products. This is not a matter of tone; it is a structural limitation that stems from the generation objective and the evidence available.

For example, if you ask about the name of a hypothetical, unreleased GPU as if it were a real product, the model may infer patterns from the name and invent a memory capacity and release date. In programming, it may confidently present a function name that does not exist, and in legal questions it may mix past and current provisions. The fact that such an answer is grammatically smooth and even includes a table is not evidence of accuracy. The greater risk is automation bias, where people feel reassured by polished wording and skip verification.

Design where failures are caught instead of merely asking the model never to be wrong. Ground current facts in source retrieval and citations, recalculate numbers, validate structured outputs with JSON schemas and test code. Require human approval for hard-to-reverse actions such as deletion, payments and external sending. RAG can retrieve wrong documents too, so separately evaluate source dates, versions, permissions and whether answers faithfully follow evidence.

Why does this happen?

Treating a model as a probabilistic generator rather than a fact database lets you plan verification costs that match the risk.

When is it a problem?

If you rely only on the prompt “If you are unsure, say you don't know,” you will miss boundary cases in production where the model is confidently wrong.

Common beginner misconceptions

Adding RAG does not eliminate hallucinations. Retrieval can miss or the model can ignore the evidence, so check the retrieval and generation stages separately.

How to verify it yourself

Mark dates, numbers, proper nouns and citations in the answer, then compare each against the source. If no source is provided for checking, classify the sentence as a claim awaiting verification, not a fact.

A local LLM (local large language model) is a setup in which model computation runs on equipment the user controls, such as a personal PC, workstation, or company server. It works without an internet connection, makes it easier to avoid sending sensitive source text to external model APIs, and keeps the cost of repeated calls predictable within the equipment cost. Pinning model and runtime versions also makes it easy to reproduce the same environment. However, apps can still communicate through update checks, telemetry, external search, or extensions, so you can say "nothing is sent externally" only after checking the actual network traffic.

Operational responsibilities that the cloud provider handled return to the user. You must manage the provenance of models and executables, licenses, malicious-file checks, GPU driver and runtime compatibility, accounts and access control, personal data in chat logs, backups, and failure recovery. A setup used by one family member on their own PC and a setup that multiple employees access through an API are not the same local service. The latter needs Transport Layer Security (TLS, encrypted communication), authentication, request limits, audit records, and resource isolation.

For example, running hospital document summaries on an in-house GPU does not protect privacy if every employee can view the source documents and prompt logs. Conversely, an individual polishing a public blog draft may face low data risk even when using an external API. Rather than treating local or cloud as inherently good or bad, compare data sensitivity, required quality, response speed, expected request volume, hardware and power costs, operating staff, and tolerable downtime in the same table.

Why does this happen?

Consider computation location and security controls separately so you do not overlook the actual data paths and responsible owners.

When is it a problem?

If authentication and logging policies are skipped just because a system is local, other users on the same network can access sensitive conversations or the model API.

Common beginner misconceptions

Open-weight does not mean free, unrestricted or safe. Check licenses, usage restrictions and deployment-file provenance separately.

How to verify it yourself

While the model runs, check the operating system's connection list and firewall logs, and document in a data-flow diagram where inputs, outputs, retrieved documents, and error logs are stored and for how long.

Small models are useful for clearly defined inputs and outputs: document classification, summary drafts, standardized writing, simple code explanations and unit-test drafts, subtitle cleanup and narration scripts. Classifying customer inquiries as payment, shipping, exchange, account or other lets people quickly review results and correct misclassifications. Extracting candidate dates and owners from meeting minutes with source links is also relatively easy to verify. Define success using measurable accuracy, omission rates and review-time savings rather than whether the model looks intelligent.

When creating narration for Text-to-Speech (TTS), an LLM may refine the script rather than act as the speech model that generates pronunciation. Calling Speech-to-Text (STT), LLM summarization, and TTS a single AI makes it difficult to locate errors. Incorrect transcription of meeting audio and factual omissions by the summarizer require different fixes. Dividing the workflow into stages makes it easier to combine small models with conventional programs appropriately and recover from failures.

But it's not the first project to ask for the latest expertise without a foundation, to put in very long documents at once, to come up with multiple systems, and to fully automate complex decisions. The bigger the loss or the harder it is to figure out the correct answer when the model is wrong, the bigger the model, the RAG, the dedicated search and settlement tools, and the approval process. If the small model is enough within the scope of the task, it's faster, cheaper, and easier to operate than choosing the larger model.

Why does this happen?

Narrowing the task scope lets you concretely define the required model size, data, evaluation items, and failure handling.

When is it a problem?

If you start by targeting an assistant that handles all company work, you cannot define correct-answer criteria or measure which model has improved.

Common beginner misconceptions

It is wrong to conclude that small models are useless, and equally wrong to conclude that every simple task can safely be delegated to them. The task and whether its output can be verified determine success or failure.

How to verify it yourself

Have people first process 20 candidate task cases and record correct answers and time spent. Compare model accuracy, omissions and review time under the same criteria.

An 8B in a model name means roughly 8 billion parameters (numbers adjusted during training). Stored at 16 bits per weight, the simple theoretical size is 8 billion × 16 bits ÷ 8, about 16GB. Applying 4-bit quantization reduces this to about 4GB, but actual files also contain quantization scales, metadata, and some values at other precisions, so they can be larger than the theoretical value. This calculation is only a starting point and does not lead directly to the conclusion that “a 4GB file runs on a 4GB GPU.”

During generation, the model keeps a Key-Value cache (KV cache, the space that stores computations for earlier context) so it does not recompute intermediate attention results for previous tokens. The longer the context and the more concurrent users, the larger the cache. Memory must also be left for the runtime's compute buffers, graphs, the driver, and Video Random Access Memory (VRAM, GPU-dedicated memory) used for display output. So rather than loading a model that uses 11.5GB on a 12GB GPU, it is safer to start around 8–9GB and measure with the actual longest inputs and concurrent requests.

Out of Memory (OOM) is an execution stop caused by total demand exceeding available memory. Instead of blindly restarting, reduce context, maximum output length, and concurrent requests to establish a baseline, then adjust model size or quantization. Changing multiple values at once hides which item was the cause. A setting that fills memory to its limit is not stable: small updates or input-length changes can stop execution again, so determine headroom through repeated measurements.

Why does this happen?

Understanding memory beyond the weights explains why a model loads but stops only during long conversations.

When is it a problem?

Buying hardware based only on model file size may handle short questions, but OOM errors can occur with long documents or once a second user arrives.

Common beginner misconceptions

A 4bit model does not mean that every calculation is performed in 4 bits. Storage precision, compute precision, and KV cache precision can be considered separately.

How to verify it yourself

Record memory and speed immediately after runtime startup, with a short prompt, at the target maximum context, and with 2 concurrent requests, in that order. The increase at each stage reveals actual safety headroom.

First ask who will produce which results from which inputs, rather than which model is best. Record whether inputs contain personal data or company secrets, whether outputs are drafts or automatically executed commands, the loss caused by one error, and who can verify correctness against which original sources. Then define daily request volume, concurrent users, acceptable response times and budget. This information supports a reasoned choice of local execution, model size, context, runtime and GPU.

For example, a personal task of turning family photo filenames into descriptions is a good fit for testing a small vision-language model locally, because the owner does not want the data sent externally and can check the results at a glance. By contrast, automatically judging the legal risk of company contracts and sending the results should not begin without evidence, authorization, and expert approval. Even the same summarization task carries different data risks and access controls for public news drafts than for patient records. A single model performance table cannot make this decision.

Start by collecting 20 to 50 representative cases, having people produce the expected results, and then measuring a baseline with the smallest candidate model. Include not only normal cases but also failure cases such as empty documents, ambiguous questions, outdated regulations, and prohibited requests. Record not just accuracy and speed but also human correction time, missing evidence, and whether requests that should be refused are handled. Increase data and user counts only when the pass criteria are met. Validating at small scale before expanding reduces hardware and operating costs and limits the scope of incidents.

Why does this happen?

Tying model selection to task requirements avoids decisions driven by trendy model names or top benchmark scores.

When is it a problem?

Watching demos without success criteria leads people to mistake a few impressive answers for overall quality and to discover omissions and costs only after going live.

Common beginner misconceptions

Installing a large model first and then finding a use for it may be fun for learning, but it is not the design sequence for an operational project.

How to verify it yourself

Fill in a table with input examples, expected output, acceptable errors, prohibited behavior, verifier, processing time, and data storage location. Blank cells are requirements to resolve before installing a model.

In a document editor, the Hangul syllable “ga” appears as one character. A computer, however, first represents characters as Unicode code points (international numbers assigned to characters), and files and networks convert them with an encoding such as UTF-8 (a rule for storing those numbers as bytes). One Hangul syllable may take one position on screen but several bytes in UTF-8. The tokenizer (the component that turns text into pieces and numbers the model can process) then performs its own separate segmentation. “One character,” “one byte,” and “one token” are therefore not three names for the same amount but distinct units at different processing stages.

Visually identical strings can have different internal representations. A Korean syllable may be stored as a precomposed syllable code point or as a sequence of jamo code points for initial consonant, vowel, and final consonant. Unicode Normalization Forms standardize equivalent character sequences. Segmentation can vary with tokenizer normalization, so identical appearance of copied sentences does not establish identical token IDs.

This distinction is not merely a matter of terminology. File upload limits are set in bytes, editors display length in characters or words, and LLM context limits and usage are counted in tokens. If a customer asks, "The document is only 2MB, so why can't I add it?", explain that the size of a file containing images and formatting is not directly proportional to the token count of the extracted text. Conversely, even a small text file can use more tokens than expected when repeated code, long digit strings, or garbled OCR characters are split into small pieces.

Why does this happen?

Each stage uses different units, so to solve the problem, identify whether the size grew in the file, the character representation, or the token split.

When is it a problem?

Treating character count as token count can lead to discovering context overflows, cost miscalculations, and insufficient output space only after deployment.

Common beginner misconceptions

It is incorrect to assume that more UTF-8 bytes always mean proportionally more tokens. The tokenizer’s vocabulary and segmentation algorithm regroup and split the input.

How to verify it yourself

First record the character count and UTF-8 byte count of the same sentence, then print the token strings and IDs produced by the actual model tokenizer side by side, and compare the three columns.

The first stage, normalization, adapts source text to the representation expected by the tokenizer. Tokenizers differ in whether they apply Unicode normalization such as NFC or NFD, case conversion, whitespace handling or accent removal. Normalization helps handle equivalent expressions consistently in retrieval and training, but can change information. If product-code case matters or original spelling must be preserved, store normalized model input separately from the original text shown to users.

The second stage, pre-tokenization, first splits candidate boundaries such as spaces, punctuation, and numbers. The third stage, the tokenizer model, further divides these pieces into subwords in a vocabulary (a list of token strings and IDs) and maps each to an integer ID. Byte-Pair Encoding (BPE), which repeatedly merges frequent pairs, WordPiece, which selects useful subwords according to a training criterion, and Unigram, which chooses probable combinations among possible segmentations, are different algorithms for this stage. Their shared goal is to represent frequent expressions in larger pieces and rare expressions in smaller pieces using a limited vocabulary.

Final post-processing may add special tokens the model requires, such as sentence starts and ends or boundaries between sentence pairs. After generation, the decoder (the component that turns token IDs back into human-readable text) combines subword markers and spacing rules to restore the sentence. A greeting that looks natural on screen was not necessarily one piece internally. For accurate diagnosis, inspect the normalized text, pre-tokens, token strings, token IDs, and special-token inclusion at each stage, not just the final string.

Why does this happen?

Splitting the pipeline into stages shows whether a change in the same sentence comes from Unicode normalization, boundary rules, vocabulary, or special tokens.

When is it a problem?

Replacing only some tokenizer files can misalign ID meanings even when the model weights are the same, sharply degrading answer quality or breaking sentences.

Common beginner misconceptions

A tokenizer does more than split sentences at spaces. It includes learned subword rules and post-processing to handle languages without spaces and rare strings.

How to verify it yourself

Print tokens, ids, offsets, and the special token mask from the tokenizer's encode output, and map line by line which range of the source text became which ID.

Storing every word as a single token would make the vocabulary grow endlessly with each new word, while using single characters as tokens makes sentences far too long. Subword tokenization takes the middle path. It adds strings that frequently occur together in training data, such as “transform” and “ation”, and recurring Hangul fragments to the vocabulary, and it represents unseen words as combinations of known pieces. Common expressions are therefore processed in fewer tokens, while rare proper nouns, long URLs, and misspellings can split into more tokens.

BPE starts with small units and learns merge rules for frequent neighboring pairs in training data. WordPiece and Unigram train differently and may produce different boundaries for the same sentence. SentencePiece can train subword models on raw sentences without assuming pre-segmented words. The key conclusion is not that one algorithm is always superior, but that inference must use the exact vocabulary and rules of the tokenizer used to train the weights.

For example, if the internal product code "KDX-2026-α" rarely appeared in training data, its hyphens, digits, and Greek letter may be split into several tokens. If this code repeats hundreds of times per document, it consumes context quickly, and even small notation differences can change search embedding results. Yet simply adding the product code to the production tokenizer as a single token does not solve the problem. Unless the embedding for the new ID is trained or the model is adjusted, that ID has no prepared meaning, so first consider whether external handling such as normalization, alias dictionaries, or search metadata can solve it.

Why does this happen?

Subword tokenization is a tradeoff that represents new strings with a limited vocabulary while shortening sequences for common expressions.

When is it a problem?

Applying ordinary-prose character/token ratios to logs, code or OCR documents with repeated rare strings can badly misestimate input budgets.

Common beginner misconceptions

One meaning does not imply one token. Token boundaries depend on learned string frequencies and algorithms rather than a semantic dictionary.

How to verify it yourself

Pass ordinary words, internal proper nouns, typos, URLs and UUIDs through the same tokenizer, compare counts and boundaries, and calculate total usage across repetitions.

Korean attaches particles and endings to stems to express grammatical roles, honorifics and tense. Forms meaning “find,” “found” and “did you find?” are related but have different surface strings. Some tokenizers store frequently seen spacing units as large tokens; others split stems and endings or smaller character fragments. English also splits words such as run, running and unpredictability differently, so one English word is not always one token. These differences reflect compression efficiency from training data and vocabulary design, not superiority of one writing system.

Tokenizers trained predominantly on English may store common English words as long tokens while equivalent Korean expressions split into several pieces. Tokenizers trained with sufficient multilingual or Korean material may use larger Korean units. A fixed claim such as 'Korean costs exactly twice as much as English' can become wrong when the model changes. Translation length, style and mixtures of numbers and English product names also matter; do not generalize one example to the whole task.

Compare representative sentence sets with equivalent meanings. Prepare short conversations, official documents, technical documents, table extracts and customer inquiries with inconsistent spacing in Korean and English, and measure token-count distributions for exact tokenizer revisions. Inspect the 95th percentile and largest increases as well as means to set safe context and quota limits. Translation to reduce tokens can lose source meaning and adds translation cost; remove unnecessary boilerplate, duplicate documents and old conversations first.

Why does this happen?

Because word forms and tokenizer training distributions differ by language, the number of long pieces the vocabulary can reuse differs even for the same character count.

When is it a problem?

Testing context budgets only with English examples can cause Korean customer inquiries or mixed technical documents to be truncated first in production.

Common beginner misconceptions

Neither one Korean syllable always equaling one token nor one English word always equaling one token is a model-independent rule.

How to verify it yourself

Measure 50 anonymized Korean, 50 English and 50 mixed-language task sentences with the actual tokenizer. Store mean, median, 95th percentile and maximum by revision.

A person may read “stock: 1000 items” and “stock: 1,000 items,” or “2026-08-26” and “August 26, 2026,” as the same information, but to a tokenizer they are different strings with different combinations of commas, hyphens, spaces, and digits. Strings that follow a pattern but rarely repeat, such as long serial numbers, hashes, Base64, and UUIDs, can be split far more finely than short natural-language text. This is why pasting just a few lines of logs into a prompt can use up a large part of the context. A better design passes only the important values in structured form and retrieves the full original when needed.

An emoji may look like a single image on screen, but it can combine several Unicode code points. When skin tone, gender, or family combinations, variation selectors, and zero-width joiners are involved, character-counting methods and token splits differ. Broken characters and control characters from OCR errors are even worse. Users see one odd symbol, but the tokenizer may process it as several byte-level pieces, and search may fail to find the same word. If characters removed during input cleaning are not recorded, it becomes hard to trace positions in the source text and answers later.

Whitespace and line breaks matter for code syntax or readability, so do not indiscriminately collapse them as natural language. Indentation, long identifiers, import paths and repeated generated code consume tokens, but arbitrary compression can change execution semantics. First select relevant functions and the area around errors, preserve line numbers and file boundaries, and exclude files unnecessary for the question, such as build outputs, lockfiles and minified code. Reducing scope while preserving source structure is more accurate than deleting characters merely to fit a token count.

Why does this happen?

A tokenizer processes actual strings and learned boundaries, not units of meaning that people recognize, so spelling differences directly produce different input arrays.

When is it a problem?

Inserting logs, OCR output, code, and identifiers without cleanup consumes context faster than natural-language documents and makes search results unstable.

Common beginner misconceptions

Assuming that whitespace and symbols carry no meaning and can all be deleted is dangerous. You can lose code syntax, table columns, document positions, and proper-noun boundaries.

How to verify it yourself

Write the same meaning as four strings that differ only in number·date·whitespace formatting, compare their token boundaries, and check that positions in the original can still be traced before and after cleanup.

Even if the chat UI shows only the final one-line question, the application may assemble system instructions, safety rules, user and assistant roles, earlier conversation, retrieved documents, tool schemas and execution results into one request. A chat template converts roles and content into the control-token format seen during model training. Chat models derived from the same base can use different templates, so copying another model’s template can degrade role separation and stopping behavior. Counting only visible characters misses all these hidden inputs.

With tool calling, function names, descriptions, and parameter schemas can consume hundreds or thousands of tokens per request. RAG documents may be much longer than the question, and multi-turn agents reinsert observations and tool outputs at each step. A quota such as “500 characters per question” therefore cannot protect actual GPU memory and throughput. At the gateway or runtime, measure final prompt tokens, reserved output tokens, cached tokens, and actual generated tokens separately.

Check immediately before actual transmission, not at the UI input box. After assembling the messages array and tools, the application applies the same tokenizer and chat template to count tokens. If the input limit is exceeded, explain which elements are large before calling the model. Even when recovery automatically summarizes old conversations, adjusts retrieved-document count, or reduces the tool list, record source-information loss and permission changes. Simply cutting off the beginning may remove system rules or the question’s premises.

Why does this happen?

A chat model does not read the role dictionary directly; it reads the token sequence the template produces, so invisible control tokens and additional inputs also consume context.

When is it a problem?

Measuring only the user's question cannot explain why a short question still exceeds the context or leaves no room for output.

Common beginner misconceptions

When earlier conversation is collapsed on screen, it is easy to assume it has also disappeared from the model input, but if the app resends it, all of it is included in the current request.

How to verify it yourself

Record an anonymized list of messages, tools, and retrieved documents immediately before the API call. After apply_chat_template, output the length including special tokens and each component’s share.

A Context window is the token range a model can handle in one request. Some runtimes limit input plus new output and may also impose separate max-input/max-output limits, so check official documentation and actual errors. An 8,192-token limit cannot fit 8,000 input tokens plus a requested 1,000-token answer. Unless the application specifies what is truncated or rejected, important content can disappear at different positions for different users.

When building a budget, first reserve the maximum output the task needs. Then measure current values in this order: fixed system prompt and template, current question, conversation history, and RAG documents. For example, reserving 1,200 tokens for output, 500 for system and template, and 300 for the question within an 8K limit leaves about 6,000 tokens for history and retrieved documents. Using all of it without a safety margin means even small changes in tokenizer revision, tool schemas, or document length can exceed the limit.

When the limit is exceeded, do not mechanically delete tokens from the beginning. First summarize older conversation into facts, decisions and unresolved items, reduce RAG chunks unrelated to the question, and remove duplicate system instructions and tool descriptions. If conditions and conclusions are far apart in a large document, preserve headings and sections and improve retrieval and reranking rather than indiscriminately shrinking chunks. After recovery, rerun the same longest questions to test answer omissions and citation accuracy.

Why does this happen?

The generated answer continues in the same sequence, so fitting only the input within the limit does not leave room for the complete request.

When is it a problem?

Without an output reservation, the answer can be cut off midway, leaving the JSON unclosed or dropping citations and conclusions.

Common beginner misconceptions

It is wrong to assume that raising the context limit always improves quality. Processing time and KV cache grow, and key evidence can be buried in long input.

How to verify it yourself

In the lab, add up the system, history, document, question, and output reservation budgets, bring an over-budget state under the limit by reducing only history and documents, and then run a regression test with the actual longest input.

During planning, no model may be selected yet, so rough ranges can help. Character-type heuristics can estimate minimum and maximum ranges, but must not serve as purchase guarantees or operational limits. The same sentence might produce 30 tokens in model A and 46 in model B, with chat templates widening the difference. Estimation tools identify relatively risky sentences; they do not imitate the authoritative tokenizer.

In the deployment checklist, record the model ID and revision, the tokenizer file's commit or digest, chat template, special-token inclusion, truncation and padding settings, and measurement code. Include not only ordinary sentences but also empty inputs, very long Korean text, technical documents mixed with English abbreviations, digit strings, emoji, OCR errors, code, and tool schemas. Recording the mean token count, 95th percentile, maximum, and failing source types lets you set quotas and chunk sizes on evidence.

When updating a model or tokenizer, measure again with the same input set. Fewer tokens do not mean better quality, and existing prompts may be split at different boundaries, which can change answers. Compare not only whether context overflows decreased but also retrieval recall, format compliance, preservation of Korean proper nouns and numbers, first-token latency, and KV cache. If something goes wrong, you must be able to roll back the previous tokenizer, template, and model as one set to isolate the cause.

Why does this happen?

Rules of thumb do not know the actual vocabulary or template, so they can only give a range; operational decisions require reproducible, actual ID arrays.

When is it a problem?

If only the model weights are pinned and the tokenizer or template is left on a moving version, the token count and meaning of the same prompt can change without warning.

Common beginner misconceptions

A tokenizer that produces fewer tokens is not always better. Evaluate training compatibility, language quality, special-character handling, and model performance together.

How to verify it yourself

Keep fixed evaluation inputs in JSONL and automatically compare tokens, IDs, totals, and output quality before and after deployment. Put promotion on hold if differences exceed permitted limits.

A tokenizer is the contract that maps displayed strings to model embedding row IDs. Attaching a tokenizer with a different vocabulary order to existing weights makes the same ID point to a different piece, which can cause meaningless repetition, exposed special tokens, and termination failures. Even when vocabulary sizes happen to match, ID mapping, normalization, and pre-tokenization rules can differ. Download the tokenizer files, config, and special tokens from the model repository at the exact revision and pin them in one manifest with the weights.

Chat templates are part of tokenizer migration. Changes in Beginning of Sequence (BOS), End of Sequence (EOS), role, and tool-control token IDs or insertion points affect both token count and conversation interpretation. Do not immediately label shorter responses after a template change an efficiency gain. Retest system compliance, multi-turn roles, tool schemas, termination, and output format on the same dialogue set.

Before replacement, prepare a fixed corpus covering Korean spacing, particles, proper names, English abbreviations, numbers, dates, emoji, OCR errors, code, and tool schemas. Compare normalized text, token strings and IDs, totals including special tokens, and upper percentiles between the previous and candidate tokenizers. Changed IDs may be normal, but explain whether associated weights match the candidate and whether context quotas, chunk boundaries, and costs remain within limits.

If context-limit errors or repeated outputs increase after promotion, roll back the model/tokenizer/template bundle to the previous revision rather than editing only the new tokenizer. Verify recovery of the same failing inputs, then change just normalization, template or quota and retest once the cause is isolated. Report averages plus maximum increases in Korean, numeric and code subsets, context-limit error counts and output quality so costs affecting only some users are visible. Preserve cases where customer names or numeric units split into many more pieces as regression inputs and remeasure them on new models. Retain tokenizer JSON, model, special-token settings and digests together in rollback artifacts to reproduce the same ID arrays.

Why does this happen?

Token IDs select embedding rows and control tokens mark conversation boundaries, so the tokenizer, model weights, and template must be compatible with one another.

When is it a problem?

If changing only the tokenizer causes repetition, termination failures, token-count surges, or changes to RAG chunks, inspect ID, normalization, and template differences.

Common beginner misconceptions

Matching vocabulary size or model family name does not mean any tokenizer can be attached and yield the same inputs and quality.

How to verify it yourself

Compare normalized text, tokens, IDs, and totals for a fixed multilingual corpus between previous and candidate versions. Retest failure recovery by rolling back the model and template bundle.

A parameter is a number that changes during neural network training to reduce prediction error. The B in a model name stands for billion, so 3B means about 3 billion, 8B about 8 billion, and 70B about 70 billion parameters. These numbers are not laid out side by side in one giant table. They are distributed across many tensors: the embedding that turns token IDs into vectors, attention that computes contextual relationships, the Feed-Forward Network (FFN) that transforms each token representation, normalization, and the output projection. A tensor can be understood as data that arranges numbers of the same kind across multiple dimensions.

Beginners should first discard the idea that one parameter stores one piece of knowledge. Even the relationship that Seoul is South Korea’s capital is not retrieved from one particular number: distributed representations learned across many language contexts and computation across layers work together to produce the answer. Deleting one parameter does not neatly remove one fact. Nor do 8 billion parameters mean 8 billion verified facts. Knowledge accuracy, freshness and source attribution require separate evaluation and retrieval systems.

Parameter count is useful for estimating capacity and compute. For two models with similar architecture and precision, the one with the larger number tends to require more weight storage and memory bandwidth. However, the actual file reflects quantization, some tensors at other precisions, and metadata, and execution also needs KV cache and workspace. B is therefore a first clue about a candidate's rough scale, not a final value that decides downloadability, speed, and task acceptance all at once.

Why does this happen?

Reading B correctly makes it possible to estimate model file size and keeps performance comparisons from overlooking conditions other than size.

When is it a problem?

Reading 70B as a model 8.75 times more accurate than 8B leads you to buy more hardware than you need or to deploy a large model without evaluating it on real work.

Common beginner misconceptions

Parameters are not human-readable fact rows or dictionary entries. They are learned values distributed across many computational matrices.

How to verify it yourself

In the candidate model’s config and weight index, find the names and shapes of the embedding, attention, and FFN tensors, and note which components hold the numbers.

More parameters may increase the capacity to represent complex patterns, but whether the model makes good use of that room depends on training. Data language and quality, total training tokens, duplication and contamination, optimizer and learning rate, architecture, instruction tuning, and safety tuning all change the result. Google DeepMind's compute-optimal research showed experimentally that, under a fixed compute budget, training tokens should be scaled along with model size instead of increasing model size alone. This result is not a rule to apply a specific ratio to every model forever; it is evidence that size alone cannot determine how well a model has been trained.

Classifying Korean parcel-delivery inquiries into five categories has a clear output format and easily checked answers. An 8B model tuned for Korean and the task may achieve higher format compliance than a general-purpose 70B. A larger model may help with conflicting conditions across documents or complex program-state changes. These outcomes are consistent: define model quality in terms of task, prompt, evaluation metrics and acceptable cost.

A fair comparison uses the same model variant, tokenizer, chat template, precision, input and output limits, and evaluation set. Read public benchmarks under the reported shot count, prompt, language, scoring method, and model revision. For your task, measure accuracy, evidence fidelity, format compliance, first-token latency, generation speed, peak memory, and failure recovery. Start with the smallest passing candidate to reduce costs and scale up incrementally if quality is insufficient.

Why does this happen?

Only a task-based comparison can explain why public rankings differ from real users' success rates and support reproducible choices.

When is it a problem?

Directly comparing scores produced with different prompts and quantization in one table cannot separate the effect of size from the effect of settings.

Common beginner misconceptions

Saying that larger models tend to have an advantage is not the same as saying a larger model is best under every task and cost condition.

How to verify it yourself

Fix 20 normal, 5 boundary, and 5 failure cases from your task. Run both candidates with identical settings and record quality, latency, and memory together.

When the tokenizer turns “I revised the report” into token IDs, the model does not perform arithmetic directly on those IDs. It retrieves the corresponding rows from a vocabulary-size × hidden-size embedding matrix to obtain vectors. With a hidden size of 4,096, each token position is represented by 4,096 numbers. These coordinates are not human-assigned meanings; they are adjusted together during training. Tokens frequently used in similar contexts may show similar relationships, but input embeddings alone do not determine their final meaning in a sentence.

Sentences have order, so position must also be represented. “The administrator approved the user” and “The user approved the administrator” use the same tokens but have opposite subjects. The original Transformer paper used sinusoidal positional encoding, and modern causal models may use methods such as Rotary Position Embedding (RoPE). Raising only the context number at runtime while ignoring the position scheme and training range can leave the model unable to use distant positions properly or degrade quality.

Combined embedding and positional representations change through attention and FFN layers. Even with the same token ID for “she,” preceding text such as “Minji revised the report” can produce a representation linked to Minji, while another context may refer to someone else. Do not overstate embeddings as human-readable coordinate tables. Research tools inspecting layer hidden states on identical sentences offer clues to relationships, not complete proof that the model answered solely for that reason.

Why does this happen?

Distinguishing input IDs, embeddings, and contextual hidden states lets you diagnose tokenizer mismatches and model reasoning problems at different layers.

When is it a problem?

Connecting a different tokenizer to the same weight can misalign the embedding rows referenced by token IDs, causing repeated characters, meaningless output, or failure to terminate.

Common beginner misconceptions

Each number in an embedding does not carry a single human-readable meaning, and there is no guarantee that nearby vectors mean the same thing in every context.

How to verify it yourself

In the deployment artifact, check that the tokenizer revision, vocab size, and special token IDs match the config's vocab_size, and compare token IDs for representative prompts with the previous deployment.

A simplified Decoder-only Transformer block can be viewed as normalization, masked self-attention, residual addition, normalization, FFN, and residual addition, in that order. Masked self-attention hides positions after the current one so the model cannot see upcoming tokens, and computes relationships with the preceding context. The FFN does not mix positions directly; it expands each position's state into a wider intermediate dimension and projects it back to the hidden size. Modern implementations can differ in normalization placement, activations, and attention variants, so the model architecture documentation is authoritative for the actual wiring.

Residual connections add block inputs to new computation results. Deep networks accumulate necessary changes without overwriting all previous information at each stage, and training signals can pass more easily across layers. Normalization stabilizes computation by controlling value distributions and scales; it is not a database of facts. Claims such as “attention stores all knowledge” or “FFN is knowledge, attention is reasoning” oversimplify intertwined computations.

Structure also explains failure symptoms. Mismatches between checkpoint tensor shapes and configuration hidden size, intermediate size or layer count produce loader size errors. Incorrect weight-name mapping during conversion can destroy output quality even if files open. Do not fill missing tensors arbitrarily during recovery: restore the official source revision, architecture class and configuration, then compare representative inputs before and after conversion. The structural diagram preserves this diagnostic sequence.

Why does this happen?

Understanding the whole block helps avoid explaining an entire model from one attention visualization and makes it possible to find config and weight mismatches systematically.

When is it a problem?

Forcibly converting a checkpoint with a different architecture can make loading fail on a shape mismatch or cause a silent quality regression.

Common beginner misconceptions

The encoder-decoder diagram in the original paper does not exactly match the detailed ordering in every modern decoder-only model.

How to verify it yourself

Open the official architecture documentation and config, and record the normalization type, number of layers, hidden and intermediate sizes, and activation and attention types in one table.

Self-attention creates Query (Q), Key (K), and Value (V) vectors from each token state. As an analogy, Query represents what the current position is looking for, Key represents the clues each context position offers, and Value represents the content to retrieve when selected. Scale the Query–Key dot products by the square root of the head dimension, then apply a mask and softmax to obtain attention weights that sum to 1. Use these weights to mix Values. This analogy explains the computation order; it does not mean the vectors literally contain human-written questions.

In “Minji corrected the report. She found an error,” the position of “She” can gather information related to the earlier “Minji.” However, you cannot conclude that a single attention weight is the complete cause of how the pronoun is resolved. Multiple heads and layers operate at the same time, and the FFN and residual connections then change the state. A visualization showing that the model attended heavily to a particular token is a clue for exploration; confirming a causal explanation for the answer additionally requires intervention experiments that change the input and compare the outputs.

The causal mask also matters. If a next-token model sees future answers during training, its conditions differ from inference. Mask future attention scores so each position attends only to itself and earlier positions. Incorrect masks or padding can leak answers, make training metrics look unusually good, and sharply reduce generation quality. Verify training and inference masks, padding, and position IDs on the same representative sequences.

Why does this happen?

Understanding Q·K·V and causal masks helps you read attention diagrams accurately and distinguish training-leakage symptoms from padding errors.

When is it a problem?

If the future-token mask breaks, training loss may look low, but the model learns from information that is not available during actual autoregressive generation.

Common beginner misconceptions

A high attention weight alone cannot fully explain human-like reasoning by the model or the cause of its final decision.

How to verify it yourself

In a short sentence using pseudonyms, mark the positions visible to the current token, and verify from framework output that the upper triangular region of the causal attention mask is blocked.

Multi-Head Attention(MHA) divides Query·Key·Value so multiple heads calculate relationships in different representation subspaces. Standard MHA has equal Query and KV head counts. Multi-Query Attention(MQA) lets all Query heads share one Key head and one Value head. Grouped-Query Attention(GQA) lies between them, dividing Query heads among several KV groups. The original GQA paper introduces an intermediate configuration with more than one KV head but fewer KV heads than Query heads, aiming for quality close to MHA and speed close to MQA.

With 32 Query heads and 8 KV heads, every four Query heads share one KV group. With the same number of layers, head dimension, number of tokens, precision, and batch, the head entries stored in the KV cache are one quarter of those in MHA with 32 heads. However, total runtime memory does not shrink to one quarter. Embedding and FFN, Query and output projections, the rest of the model weights, and runtime workspace do not disappear. A conclusion such as “because it uses GQA, a 70B model runs in 8B memory” is wrong.

When reading a model config, write down num_attention_heads and num_key_value_heads separately and check the head dimension as hidden_size÷num_attention_heads. If the Query head count is not divisible by the KV head count, the model may not use simple equal-sized groups, or the config may have been misread. The official Llama 3.1 model card states that 8B·70B·405B all use GQA, but specific tensor shapes and runtime support must be verified against that model's config and implementation.

Why does this happen?

Read the KV-head count correctly to avoid overestimating or underestimating cache memory for long contexts and concurrent requests.

When is it a problem?

Substituting Query head count into the KV-cache equation overestimates GQA cache. Applying the GQA ratio to the entire model instead underestimates total memory.

Common beginner misconceptions

Do not assume that each head is dedicated to grammar, facts, or reasoning, or that GQA reduces the number of Query heads itself.

How to verify it yourself

Find the two head counts in the official config and calculate the group size, then record the difference between the estimate and the cache memory the runtime reports at the same context.

For a beginner's approximation, calculate embedding parameters as vocabulary size × hidden size. If the output projection uses weight tying (sharing weights) with the input embedding, this table is reused; otherwise an output matrix of similar size is added. Attention sums the shapes of the Q, K, V, and output projections, and gated FFNs often use three large hidden × intermediate matrices, depending on the implementation. Multiply by the number of layers and add small items such as normalization to see the approximate breakdown of total model size.

The purpose of this calculation is not a game of reverse-engineering the official parameter count; it is to understand which setting drives which cost. Increasing the vocabulary enlarges the embedding and, in some cases, the output table; increasing hidden size sharply grows several square projections; intermediate size directly enlarges the FFN; and layer count repeats the block cost. The formula also shows that reducing KV heads shrinks the K and V projections and the cache, while the Query, output, and FFN stay the same.

Actual architectures differ in biases, gated-activation matrix counts, tied embeddings, local/global attention, mixtures of experts and multimodal projectors, so they do not exactly match the simple formula. Label exercise results educational approximations and verify against official model cards, configurations and sums derived from weight-index tensor shapes. Large discrepancies mean you must explain omitted structure rather than declare the model wrong.

Why does this happen?

Breaking the total parameter count in billions into components explains why changes to hidden dimensions, FFN and heads affect capacity and cache differently.

When is it a problem?

Counting only two matrices for a gated FFN or omitting an untied output can substantially underestimate size and lead to an incorrect equipment budget.

Common beginner misconceptions

Educational approximations do not replace official parameter counts for every architecture, and matching a number does not prove implementation compatibility.

How to verify it yourself

In the first lab, calculate the share of each component, compare it with the sum of tensor shapes in the official weight index, and record the names of missing items.

When reading a model card, do not take only the size from its first line. Check the developer and release date, pretrained/base versus instruction-tuned status, text-only versus vision input, supported languages and knowledge cutoff, architecture, context, intended use and known limitations, evaluation methods, license, and acceptable use. Compare vocab·hidden·intermediate sizes, layers, Query·KV heads, positions, and dtype in the config. If the documents disagree numerically, first verify that you are reading the exact same variant and revision.

Meta's official Llama 3.1 model card specifies text-model sizes of 8B·70B·405B, a 128K context, and GQA across all sizes. This claim supports structural information about that public model family, but it does not guarantee accuracy on your Korean-language tasks, the quality of quantized files, latency at 128K, or whether a consumer GPU can run the model. Google's Gemma 3 model card likewise distinguishes input modality and context conditions by size. Do not assume that every size in a family with the same name has the same features and limits.

The deployment record keeps the exact model ID and commit or revision, weight hash, tokenizer and chat template, quantization and dtype, runtime and driver, context and batch, license review date, evaluation set results, and rollback artifact. If output starts repeating after an update, diff which of model weights, tokenizer, template, or runtime changed, and restore the previous bundle. With this record, “why did the same 8B change?” can be explained with reproducible evidence instead of guesses from the name.

Why does this happen?

A model is not a single weight file but a bundle of artifacts that includes input formats and execution conditions, so pin them together to make comparison and rollback possible.

When is it a problem?

Recording only a mutable latest makes it impossible to reproduce the same experiment after upstream changes and can miss license, template, and config changes.

Common beginner misconceptions

The official maximum context and benchmarks are values reported under specific conditions, not a guarantee of the same results across all languages, runtimes, and precisions.

How to verify it yourself

For each candidate, record model ID, revision, hash, variant, context, tokenizer, template, license, runtime and evaluation results in one row. Do not promote candidates with missing fields.

A language model has an input embedding that converts token IDs into hidden vectors and an output projection that converts the final hidden state into vocabulary logits. The two matrices can have corresponding shapes, such as vocabulary size × hidden size. Weight tying is a design in which the output projection shares the same parameters as the input embedding. There are two roles but only one stored and trained tensor, so counting it twice in a simple calculation overestimates the total.

Config fields such as `tie_word_embeddings` are clues, but not every architecture uses the same names and storage scheme. Use the framework loader and official architecture documentation to check whether the input and output names in the weight index refer to the same storage and whether the checkpoint includes a separate output tensor. If the byte total of the Safetensors shards cannot fully explain parameter sharing, compare the loaded model's named parameters against data pointers and the official count.

Parameter sharing is not limited to embeddings. Some architectures reuse weights across layers or conditionally attach experts·adapters. Conversely, multimodal models can add a vision encoder, projector, and audio components, giving them more tensors than the text model name suggests. Do not force the teaching dense-decoder approximation onto every checkpoint; explain differences between the estimate and the official count with a list of structures.

Check that weight sharing survives conversion and quantization. If a converter materializes a shared tensor twice, the displayed parameter count may look the same while artifact bytes and load memory grow; conversely, alias handling may show only one entry in the file list. Compare tensor names, shapes, dtypes, and storage bytes between the original and converted files, then retest quality on the same prompts and peak memory to close the conversion contract.

Why does this happen?

Parameter counts measure independently learned tensor elements, not computational roles. Count shared weights only once even when used in several places.

When is it a problem?

Always counting embedding and output weights twice, or breaking weight sharing in a converted artifact, makes parameter, file, and memory estimates diverge sharply from official figures.

Common beginner misconceptions

Identical shapes do not automatically mean identical weights, and a single sharing flag in the config does not prove every artifact alias.

How to verify it yourself

In the official config, weight index, and loaded parameters, check the names, shapes, and storage sharing of the input and output tensors, and compare bytes and peaks before and after conversion.

Storing 8B parameters in 4bit gives a simple lower bound of about 4GB for the weights, but scale·metadata and mixed tensors add to that, and the runtime uses graph·kernel workspace and an allocator. Even with a fixed parameter count, activation and KV cache vary with input·output tokens, layers, hidden·KV heads, batch, and concurrency. The conclusion “the 8B Q4 file is 6GiB, so it is safe on an 8GiB device” ignores the runtime state of the longest request.

During inference, prefill activations can create temporary peaks depending on the kernel and implementation, while KV cache grows with tokens and sequences during decoding. Static cache may reserve maximum space, and paged runtimes have block and allocator overhead. This explains why one 4K-context request and four 32K-context requests have different peaks for the same model parameters. Measure GPU and system memory separately during loading, prefill, decoding and each target concurrency level.

Training stores weights, gradients, optimizer states, and activations needed for backward passes. Adam-family optimizers may keep several states per parameter, and mixed-precision training may retain master weights. LoRA and QLoRA can reduce trainable parameters and weight memory, but do not eliminate activations, quantization metadata, or optimizer states. A parameter-count label alone does not prove that hardware capable of inference can also support training.

Use separate operational columns for artifact bytes, loaded weights, loading/prefill/decode peaks, KV cache, host offload and training peaks. Attach measurement time, exact model/runtime, input/output tokens and concurrency to each value. Purchasing tables should include actual maxima for representative/longest inputs and single/target user counts, not just weight lower bounds. For offloaded layers, record transfer latency, CPU usage and response-latency percentiles alongside device-memory reduction. Mark unmeasured cells unverified, not zero. Have another operator check inputs and units before approval. Change runtime, context, batch and quantization individually and retest peaks, latency and quality on the same failures. On OOM, identify the growing memory category and first restore admission limits and previous settings rather than immediately choosing a smaller parameter name.

Why does this happen?

This is because weights are fixed tensors, while activation·cache·gradient·optimizer states are created and grow with the purpose of execution and the requests·batch.

When is it a problem?

If file and weight calculations are correct but OOM occurs during prefill, concurrent requests or training, the separate accounting for runtime state and workspace is likely missing.

Common beginner misconceptions

Parameter×bit÷8 is not all model execution·training memory, and total peaks do not necessarily shrink exactly in proportion to quantization.

How to verify it yourself

For the same artifact, measure device·host peaks during load·prefill·decode·concurrency and training, and explain the difference from the weight lower bound item by item.