The engineering techniques behind real-world LLM deployment

Training a large language model (LLM) can cost millions of dollars, and deploying one at scale can cost millions more. Despite this, the raw model straight out of training is often the wrong tool for any specific job.

This is the gap that AI engineering fills. The techniques described in this blog are the ones that turn expensive research artifacts into useful products that you use daily. These include fine-tuning a model for your specific domain without retraining it from scratch, getting a model to cite real documents instead of hallucinating (though that problem is far from solved), and running a billion-parameter model on your phone.

The architecture of transformers (covered in Part 2 of this series) provides the raw capability. What we cover here determines whether that capability becomes reliable, affordable, and useful for both specialized tasks and day-to-day AI assistance.

This is the final installment in our three-part series, and it covers key concepts that range from fine-tuning strategies to deployment challenges models face today. Each section is written to give you a working knowledge of how LLMs operate today.

Fair warning: With the pace of AI development, this blog will probably be outdated in the next 1 – 2 years.

The starting point for everything

A Foundation model is a large model trained on broad data that is used as a starting point for many downstream tasks. The term was coined by Stanford researchers in 2021 to describe a shift in how AI systems get built. Instead of training a new model from scratch for each task, you start with a pretrained foundation and adapt it.

Foundation models come in several varieties. Language models like GPT-4 and Claude handle text. Vision models like DINOv2 handle images. Others generate entirely new content, the way DALL-E produces images from text descriptions. And multimodal models like CLIP blur the lines, working across text and images simultaneously.

Training a frontier language model from scratch can require months of compute on thousands of GPUs, costing tens or hundreds of millions of dollars. Adapting an existing foundation model to a specific task might take hours on a single GPU, costing dollars. This asymmetry means that foundation models have become shared infrastructure, with organizations building specialized capabilities on top of models they did not originally train themselves.

The risk, which any honest practitioner should acknowledge, is concentration. If most AI applications depend on a handful of foundation models from a handful of companies, then bugs, biases, or policy changes in those models ripple through entire industries. Open-source models like Llama and Mistral provide alternatives, but today the majority of commercial AI applications still trace back to a small number of base models. The dependency is real.

What changed between GPTs and why it matters to most models

GPT-3 arrived in 2020 with 175 billion parameters and demonstrated that scale alone could produce fascinating capabilities. Few-shot learning, coherent long-form writing, and basic reasoning emerged from scaling up the same transformer architecture, and the AI field exploded.

GPT-4, released in 2023, changed what the model could take as input. Where GPT-3 was text-in, text-out, GPT-4 could process images alongside text, answering questions about charts, photographs, and diagrams. The context window expanded dramatically, from GPT-3’s 2048 tokens to GPT-4’s 128,000. Factual accuracy improved through better training data curation and reinforcement learning from human feedback.

From an engineering perspective, the interesting evolution was less about individual capabilities and more about reliability. GPT-3 produced impressive demos that often fell apart under sustained use. GPT-4 showed meaningfully better consistency, following complex multi-step instructions more faithfully and producing fewer obviously wrong statements. This reliability gap is what turned LLMs from impressive curiosities into a tool used in everyday business operations.

The competitive landscape shifted rapidly after GPT-4, Anthropic’s Claude, Google’s Gemini, Meta’s Llama, and Mistral’s models each pushed in different directions. The new features like longer context windows, better reasoning, open weights, and multilingual performance are used across them to enhance user experiences. Within two years, the field went from one dominant model to a crowded market where model selection became an engineering decision rather than a default.

Overfitting

Overfitting is one of the oldest problems in machine learning, and it remains relevant even at the scale of modern LLMs. A model overfits when it performs well on training data and poorly on new, unseen data. It has memorized the training examples instead of learning general patterns.

Imagine a student who memorizes every answer in a textbook word-for-word. They ace the textbook quiz, but when the exam presents the same concepts in slightly different phrasing, they fail. That is overfitting. The student (model) learned the specific examples (training data) without grasping the underlying principles.

Classical machine learning developed a toolkit for this, which included regularization techniques that penalize complexity, dropout that forces redundancy in learned representations, and early stopping that halts training before memorization sets in. While these still apply to LLMs, the more interesting overfitting story happens during fine-tuning.

Fine-tuning datasets are usually far smaller than the pretraining corpus. A model that saw trillions of words during pretraining might get fine-tuned on a few thousand examples, creating ideal conditions for memorization. This is one reason parameter-efficient methods like Low-Rank Adaptation (LoRA) have become so popular. Instead of updating all the model’s weights during fine-tuning, LoRA freezes the original parameters and injects small trainable matrices alongside them. The model adapts through these small additions rather than rewriting itself wholesale. This constrains how much the model can change, acting as a built-in guard against memorization.

LoRA also solves a cost problem. There are two ways to fine-tune a model. Full fine-tuning updates every one of its parameters. Parameter-efficient methods like LoRA update only a small fraction and leave the rest frozen. Full fine-tuning is the expensive one. For a 70-billion-parameter model, you have to hold the weights, gradients, and optimizer states in memory all at once. That runs to hundreds of gigabytes, often more than a terabyte. Few organizations have that hardware sitting around. LoRA works differently. You still load the model, but instead of changing its parameters you train a small set of new ones on top. For a 7B model that might be 10 million trainable parameters, about 0.14% of the total.

Quantized Low-Rank Adaptation (QLoRA) goes further by quantizing the frozen base model to 4-bit precision, shrinking the memory footprint of the frozen weights by about four times. Combined with LoRA’s small trainable adapters, QLoRA makes it possible to fine-tune a 70-billion-parameter model on a single GPU. The quality loss from quantization is typically minimal for most practical tasks.

The amnesia problem

When you fine-tune a model on new data, you risk destroying what it already knows. This is catastrophic forgetting, and it is a real concern for anyone adapting pretrained models. It’s also why, if you use any modern AI regularly, every new model version “feels different.” Something improved, but something else got subtly worse.

The mechanism is straightforward. During fine-tuning, the model updates its weights to perform well on the new task. If those weight updates push the model away from configurations that supported its previous capabilities, those capabilities degrade. Fine-tune a general-purpose model exclusively on legal documents, and it might become excellent at legal language while losing its ability to write poetry or answer science questions.

Three strategies address this.

1. Rehearsal (or replay) mixes examples from the original training data into the fine-tuning dataset. If 20% of each training batch contains general-knowledge examples, the model maintains those capabilities even as it learns the new domain.
2. Elastic weight consolidation (EWC) identifies which weights are most important for the original tasks and penalizes large changes to those specific weights during fine-tuning.
3. Modular architectures add task-specific components (like LoRA adapters) while keeping the base model frozen, which sidesteps the problem entirely. You can train multiple LoRA adapters for different tasks and swap them at inference time without any risk of one task degrading another.

Of the three, the modular approach has largely won in practice. LoRA eliminates catastrophic forgetting by design simply because the original weights never change so the model “feels the same.”

Teaching a small model to think like a big one

The best LLMs are too large and expensive for many deployment scenarios. For example, running a full frontier model on a smartphone is not feasible, and serving it to millions of users simultaneously is extremely costly. Distillation addresses this by training a smaller student model to replicate the behavior of a larger teacher model.

The teacher model’s soft probability outputs contain more information than hard labels. When predicting the next word in “She picked up her ___,” the teacher might output [“phone”: 0.4, “bag”: 0.3, “keys”: 0.2, “elephant”: 0.001]. The correct answer might be “phone,” but the student also learns that “bag” and “keys” are reasonable while “elephant” is nonsensical. Hard labels would just say “phone” and throw away those relationships. The “soft probabilities” encode something that is deeper: the teacher’s learned sense of what belongs in a context and what does not. “Bag” and “keys” score high because they share something with “phone” in this context. They are all objects a person picks up. “Elephant” scores near zero because nothing about the sentence supports it. The student learning from a good teacher does not only memorize the answer. It picks up the teacher’s sense of what fits, which makes it better at similar questions later.

So, what makes the student smaller? Size in a language model mostly means parameters (the learned numbers in its weight matrices) and a student simply has fewer of them. It is built with fewer, narrower layers, so it carries less internal machinery. The practical effect is that it does less arithmetic for every word it predicts, which makes it faster, and it takes up less memory, which is what lets it run, for example, on a phone or tablet.

But “smaller” can come with a real cost. A student has less room to store facts and less capacity to handle hard or unusual cases, so it will not match the teacher everywhere. Distillation helps the student make the most of the smaller budget it has, so it stays close to the teacher on the things that matter most. A well-distilled student can retain a large share of its teacher’s quality at a small fraction of the size, though how large that share is depends heavily on how broad the task is and on what you measure.

Many of the AI features already running on-device, such as autocomplete, voice transcription, and photo search, depend on model-compression techniques like distillation to shrink models that would otherwise be far too large to run locally. The tradeoff is that small models have a capacity ceiling. If the model needs to handle a wide range of tasks, you need a bigger student; if it only needs to do one thing well, you can go much smaller. Below a certain size, no amount of clever training will close the gap with the teacher. Finding the right size for a given quality target and deployment constraint is part of the engineering challenge.

Grounding AI in real documents

LLMs generate text from patterns in their training data. When they encounter questions about information not in that training data (such as your company’s internal policies, yesterday’s news, or something they just didn’t see yet), they do one of two things: refuse to answer or make something up. This is why we talk about hallucinations in AI, and some are truly wild.

Retrieval-augmented generation (RAG) solves this by connecting the LLM to an external knowledge source. The process has three steps. First, the user’s query gets converted into an embedding and used to search a document store for relevant passages. Second, the retrieved passages get ranked by relevance. Third, the top passages are included in the LLM’s prompt as context, and the model generates its response based on this provided evidence.

As a result, the AI system tries to cite real documents. Ask a RAG-powered system about your company’s parental leave policy, and it tries to retrieve the actual policy document, it includes it in context, and generates a response grounded in that specific text. You can verify the answer against the source or ask it for a source. RAG is not a silver bullet though. The model can still misread a passage, blend retrieved content with its training data or attribute a claim to a document that does not fully support it. Grounding reduces hallucinations, it does not eliminate them.

Building a good RAG system comes down to the retrieval component. This is the part that searches your documents and decides which passages to hand the model before it writes anything back to you. The model only knows what it sees in that moment, so if retrieval hands over the wrong passages, the answer will be wrong no matter how capable the model is. Good retrieval depends on how documents are broken into pieces (chunked), how the system understands the meaning of a question, how it searches, and how it decides which results are actually useful. Each of these is a quality lever, and getting them right is the difference between a RAG system that genuinely helps and one that quietly misleads. The model is rarely the bottleneck. The search behind it, and the quality of the documents it draws from, almost always are.

RAG has become the default architecture for enterprise AI applications because it addresses the two biggest concerns businesses have: accuracy and attribution of data processing. The model’s answers can be traced back to specific source documents, creating an audit trail that pure generation cannot provide right now.

Mixture of experts

Mixture of experts (MoE) is an architecture pattern that lets a model have a very large total parameter count while only using a fraction of those parameters for any given input. The model contains multiple “expert” sub-networks, and a gating mechanism selects which experts activate for each token.

Consider a model with eight expert networks and a gate that activates the top two for each input. The total model might have 100 billion parameters, but each forward pass uses only about 25 billion (the two active experts plus shared components). This means inference is much cheaper than a dense model of the same total size, while the model’s total knowledge capacity remains large. The underlying insight is that different inputs need different expertise. A question about chemistry and a question about contract law don’t need the same parameters, so why activate all of them every time?

MoE models can suffer from load balancing problems, where some experts get used heavily while others sit idle. They require more total memory even if per-token compute is lower, and distributed training requires careful routing to keep experts balanced across GPUs. Teams adopting MoE in production are likely to spend a significant chunk of their engineering effort on these infrastructure problems rather than on the model itself.

Getting AI to show its work through chain-of-thought prompting

LLMs can produce correct answers to complex reasoning problems, but they often get the wrong answer if asked to produce the answer directly. Chain-of-thought (CoT) prompting fixes this by asking the model to reason step by step before giving its final answer. Next time you ask an AI model a complex question and get a suspicious answer, try appending “Think through this step by step and use chain-of-thought” to your prompt. The difference in output quality is often immediate and obvious.

The technique is simple. Instead of asking “What is 17 times 24?” and getting an immediate (possibly wrong) answer, you ask “What is 17 times 24? Think through this step by step.” The model then breaks the problem down: “17 times 20 is 340. 17 times 4 is 68. 340 plus 68 is 408.” By decomposing the problem, the model avoids shortcuts that lead to errors.

Where this gets powerful is on problems with actual complexity. Ask a model “Should this patient be referred to a cardiologist based on these symptoms?” and a direct answer might be wrong. Ask it to reason step by step and it will work through the symptoms individually, consider which ones are cardiac-relevant, weigh the combinations, and arrive at a more detailed conclusion that can be considered by a medical professional. The difference between a one-shot answer and a reasoned chain can be the difference between a useful system and a potentially dangerous one.

CoT works because it forces the model to allocate more computation to the problem. Each reasoning step generates tokens that the model then uses as context for subsequent steps. The intermediate tokens serve as a form of working memory, holding partial results that the model can reference. Without CoT, the model must produce the answer in a single forward pass, which limits the complexity of reasoning it can perform. Smaller models do not benefit much from being asked to think step by step. Larger models, roughly 100 billion parameters and above, show significant accuracy improvements. In other words, the model needs to be smart enough to benefit from thinking harder. Below a certain size, asking for step-by-step reasoning may just produce step-by-step nonsense.

What keeps LLM engineers up at night

Getting a model to work in a research lab and getting it to work in production are very different problems. The gap between them is mostly infrastructure, where the hard engineering lives.

Resource intensity is the most obvious challenge. Serving a large model requires expensive GPU hardware, significant memory, and careful batching to achieve reasonable throughput. A single A100 GPU costs roughly $15,000 to $25,000. Serving a frontier model at scale requires clusters of these, along with the networking fabric to connect them. At Cisco, we see this firsthand. The data center infrastructure required to support AI workloads at scale is a fundamentally different design problem than traditional compute. High-bandwidth, low-latency interconnects between GPU nodes are as much a bottleneck as the GPUs themselves. The cost structure makes it difficult for smaller organizations to self-host and pushes many toward API-based access.

Latency matters for user-facing applications, and it compounds across the stack. Generating a response token by token is inherently sequential, and each token requires a full forward pass through the model. For a large model, this might take 30-50 milliseconds per token, which means a 200-token response takes 6-10 seconds. But that’s model latency alone. Add network hops between the user and the inference server, load balancer overhead, and any retrieval calls to external data sources, and real-world latency can be significantly worse. Techniques like speculative decoding, cache optimization, and model quantization help on the model side, but end-to-end latency is also a systems problem.

Privacy is often the gating concern for enterprise deployments. Models can memorize fragments of training data and reproduce them in outputs. Fine-tuned models trained on company data may leak sensitive information through clever prompting. A model fine-tuned on internal support tickets could, under the right conditions, surface a specific customer’s details. Deployment architectures need to account for data residency, access controls, network segmentation, and inference isolation. These concerns have made on-premise deployments and zero-trust AI architectures central to many companies’ enterprise AI strategies. The most frequent conversation with customers is not “which model should we use” but “how do we deploy it without exposing our data.”

From research to reality

In Part 1, we looked at where AI came from and why it accelerated so fast. In Part 2, we opened up the transformer and saw the architecture that makes modern AI possible. In this final part, we covered what it takes to make that architecture work in the real world.

The transformer itself has remained essentially the same since 2017. What changed is everything around it – fine-tuning that costs dollars instead of millions, models that cite real documents instead of inventing facts, and billion-parameter systems that run on your phone. These came from engineering, not a new architecture.

If there is one takeaway from this series, it is that engineering ingenuity matters as much as architectural innovation. The researchers built the foundation, the engineers made it work, and the gap between those two, the space where a research artifact becomes something you rely on without thinking about what’s underneath, is where the most interesting problems live right now.

If you made it through all three parts, you now have a working mental model of how modern AI systems are built, trained, and deployed. That understanding will serve you whether you are building these systems, managing teams that build them, or making decisions about adopting them. The details will change, but the fundamentals we covered might not – at least, not for a while.