Most AI projects that stall do not stall because someone wrote a bad prompt. They stall because the prompt was the only thing anyone engineered. In 2026, “the prompt is wrong” is rarely the real diagnosis. Missing context, stale retrieval, an agent with far too many tools, no approval boundary, no trace to debug from: that is what actually breaks in production, and none of it lives inside the prompt.
The cleanest way I have found to reason about this, after shipping 55+ agents into production, is to treat AI system design as three nested layers. Prompt engineering is the wording and structure of a single instruction. Context engineering is deciding what the model sees at each turn. Harness engineering is the runtime system around the model: orchestration, validation, approvals, tracing, evals, and governance. Anthropic frames context engineering as the natural progression of prompt engineering. OpenAI’s harness engineering framing extends the same line one layer out, to the full agentic system.
This piece walks each layer, shows where it breaks, and gives a build order. If you want the architecture patterns that sit underneath, read AI Agent Architecture: Reference Patterns alongside this.
The one-line version
- Prompting solved single-call behaviour.
- Context solved multi-turn cognition.
- Harnessing solves production reliability.
They are layers of responsibility, not rival techniques. You do not pick one. You build all three, in that order, and most of the reliability of a real system comes from the outermost layer, not the innermost.
A short history, so the vocabulary makes sense
The terminology here is still settling, and different teams use different labels for overlapping work. Reading it as nested layers avoids the confusion.
Prompt engineering became strategic with large-scale in-context learning around 2020, when GPT-3 style few-shot prompting showed that the instruction itself was a design surface. Chain-of-thought prompting, self-consistency, and automated prompt search followed.
Context engineering became mainstream as a named idea in 2025, but its roots are older: retrieval-augmented generation, long-context management, tool use, and memory systems. It surfaced because long-running agent workflows exposed the limit of “just write a better system prompt”.
Harness engineering is newer still as a named discipline, though its substance is familiar from orchestration runtimes, test harnesses, CI/CD, guardrails, tracing, and human-in-the-loop workflows. It emerged when teams realised reliable agents need runtime infrastructure, not only good prompts and retrieval.
Layer 1: Prompt engineering
Prompt engineering is the narrowest layer, and still foundational. The best retrieval stack or agent harness will fail if the task framing, output contract, or instruction hierarchy is poor.
At its core, a good prompt answers five questions: who the model is supposed to be, what exactly it should do, what it should not do, what output shape is required, and which examples best demonstrate the target behaviour. Separate identity, instructions, examples, and context. Pitch the instruction at the right altitude, neither brittle pseudo-code nor vague aspiration.
The design principles are simple to state and easy to skip:
- Specific rather than implicit.
- Sectioned rather than amorphous.
- Example-based where patterns matter.
- Evaluation-driven rather than intuition-driven.
One nuance worth internalising: for modern reasoning models, simple and direct prompts often beat explicit “think step by step” instructions, because those models already reason internally. Try zero-shot first, then add few-shot examples only if the behaviour is pattern-like rather than rule-like. That is a real shift from the chain-of-thought folklore of a few years ago.
The toolbox now includes zero-shot and few-shot prompting, role prompting, schema-constrained prompting, decomposition or prompt chaining, and, in some settings, automatic prompt optimisation. Academic work such as AutoPrompt and OPRO showed prompts can be searched algorithmically, and DSPy productised that idea by tuning prompts, examples, and sometimes weights against a metric. Prompt caching matters too: stable instructions, schemas, and context headers become cheaper to reuse in multi-turn workloads, which quietly changes system design.
Here is a typical extraction prompt, the kind that feels finished but is not:
# Role
You are an accounts payable extraction assistant.
# Task
Extract supplier_name, invoice_number, invoice_date, currency,
tax_amount, and total_amount from the document.
# Rules
- Return valid JSON only.
- If a field is missing, return null.
- Do not infer values that are not stated.
- If multiple totals appear, prefer the final payable amount.
# Output schema
{
"supplier_name": string | null,
"invoice_number": string | null,
"invoice_date": string | null,
"currency": string | null,
"tax_amount": number | null,
"total_amount": number | null
}Useful, but only up to a point. The moment the model also needs the purchase order, the tax policy, supplier history, duplicate-detection rules, and an approval policy, the problem has already left prompt engineering.
Where prompting runs out
Prompt wording is brittle across models and provider updates. Wording that works on one version can degrade on the next. Prompt-only systems also struggle when the real failure is missing knowledge, inadequate tools, or a safety conflict, and when prompts mix trusted instructions with untrusted third-party text, they open the door to prompt injection. Good prompt metrics are therefore multi-dimensional: task accuracy, exact match, schema validity, style adherence, refusal correctness, latency, and token cost. Score automatically where you can, and use human review to calibrate the judges. This is the same eval-driven discipline that separates the 5% of AI pilots that reach production from the 95% that do not.
Layer 2: Context engineering
Context engineering has the most momentum right now, because it addresses the central weakness of a naive agent: it is only as good as the information loaded into its context at the moment of inference. The useful framing treats context as the whole token budget available to the model, not just the user prompt: system instructions, examples, retrieved documents, tool definitions, memory, conversation history, and state carried across turns.
Five principles do most of the work.
More context is not automatically better. Larger windows allow more complex prompts, but more tokens do not guarantee better performance, and accuracy and recall can degrade as counts grow. The Lost in the Middle research makes the point precisely: long-context models do not use all positions equally, and information buried in the middle of a long prompt is often used poorly.
High-signal minimalism. Find the smallest set of high-signal tokens likely to drive the outcome. Resist giant instruction manuals, bloated toolsets, and endless edge-case examples. Expose only the tools that turn actually needs, and watch context growth over long sessions.
Dynamic retrieval over static stuffing. Add relevant retrieved context at runtime from a vector store or file search, rather than preloading everything. “Just in time” strategies, where the agent keeps lightweight references such as file paths or queries and loads data through tools when needed, are especially effective for large codebases, legal work, research, and database-backed tasks.
Memory outside the immediate window. Compaction, structured note-taking, and multi-agent designs carry long-horizon tasks. It helps to distinguish short-term thread memory from long-term cross-session memory; MemGPT formalised the same idea by treating memory as hierarchical storage tiers. Memory engineering is now a real subdiscipline.
Context isolation. Use subagents when an exploratory side task would flood the main conversation. Each subagent gets its own context window, tools, permissions, and prompt. That is context engineering by decomposition: you do not merely shrink context, you partition it.
The workflow is cyclical, not one-shot. Define stable instructions, expose a minimal toolset, retrieve or compute the smallest relevant context for the current turn, compact or summarise history when sessions run long, and carry forward only what will matter next. Common tools here include vector search, semantic retrieval, file search, memory stores, compaction functions, note files, subagents, and increasingly the Model Context Protocol. MCP matters because it standardises how tools and external systems contribute context and actions, which makes context surfaces uniform and cuts bespoke integration work. I went deep on that in How I Connected Xero to AI Using the Xero MCP Server and The MCP Server Handbook for Enterprise.
Context engineering has its own failure modes: retrieval that surfaces plausible-but-irrelevant documents, memory that goes stale or gets poisoned, large windows that dilute attention and add latency, tool outputs that blow the token budget, and untrusted external content that injects instructions into the stream. Evaluation therefore emphasises retrieval and grounding, with metrics like context recall and context precision for document Q&A, and separate scoring of retrieval steps, tool invocations, and output formatting rather than grading only the final answer.
Notice that this snippet is not a prompt at all:
context = {
"instructions": base_prompt,
"supplier_profile": supplier_lookup(invoice.supplier_id),
"purchase_order": po_lookup(invoice.po_number),
"receipts": goods_receipt_lookup(invoice.po_number),
"policy_excerpt": retrieve("invoice_tolerance_policy"),
"recent_memory": memory.search(user_id=invoice.approver, task="invoice review"),
"tools": ["lookup_supplier", "lookup_po", "raise_exception_ticket"]
}It is deciding what the model should see for this turn. That is the whole discipline.
Layer 3: Harness engineering
Harness engineering is what turns a capable model into a dependable system. If prompt engineering is about instruction quality and context engineering is about information quality, harness engineering is about system behaviour under production conditions.
A useful definition: harness engineering is the design of the execution environment around the model so that its outputs become safe, testable, observable, and operationally useful. Four principles recur.
Legibility. Make the repository, docs, tests, CI, logs, UI, and metrics legible to the agent, and encode architecture and operating rules into the system around the model. In coding workflows the repo becomes the system of record.
Explicit workflow boundaries. Define autonomy and approval boundaries up front. Human-in-the-loop workflows, tool guardrails, and resumable approvals are part of the design, not an afterthought.
Orchestration. Decide whether the model routes tasks itself, whether code routes them deterministically, or whether a hybrid is better. Patterns include agent loops, handoffs, agents-as-tools, evaluator agents that judge whether a result passes, and specialist subagents.
Measurement. Tracing, offline evals, and online monitoring are not polish. They are the harness. Without a trace for every run, you cannot debug an agent that misbehaved at 2am, and without evals you cannot tell whether a model or tool update quietly regressed you.
Runtime techniques in this layer include branching workflows, structured outputs for routing, validation passes, approval pauses, retries, circuit breakers, audit logs, and rollback paths. This is exactly the territory I mapped in AI Agent Architecture: Reference Patterns.
The harness is most of the reliability
Take an invoice-processing agent end to end. The system ingests the document, runs OCR or parsing, checks for duplicates, retrieves supplier and PO data, calls the model with curated context, validates the extraction, applies deterministic policy checks, routes exceptions to a human, records traces and audit logs, and only then posts to the ERP. The model matters, but most of the reliability lives in the harness around it.
parsed = parse_invoice(file)
if duplicate_check(parsed):
return route_to_human("possible_duplicate")
ctx = build_context(parsed)
draft = ap_agent.run(ctx)
validate_schema(draft)
validate_policy(draft, tolerance_rules)
if requires_manager_approval(draft):
pause_for_approval(draft)
post_to_erp(draft)
trace_run(draft)That is a harness artifact, not a prompt artifact. It expresses sequencing, control points, validation, and side effects. It is also, structurally, what I built when I stood up an AI accounting team over a weekend using MCP: the intelligence was the easy part, the guardrails were the work.
The major risks at this layer are complexity and blast radius. Tool misuse, approval deadlocks, hidden state coupling, thin observability, and insecure integrations can all defeat otherwise strong model behaviour. Once an agent acts on third-party content or real systems, prompt injection and tool-surface security become first-order design constraints, and frameworks like NIST’s AI Risk Management Framework and its Generative AI Profile are there precisely to force continuous measurement and residual-risk management. Harness-level metrics should go well beyond answer quality: end-to-end task completion, tool-call success rate, schema-validation pass rate, policy-compliance rate, approval-interruption rate, rollback or retry rate, cost per successful run, median and p95 latency, operator review load, and production anomaly rate from online evals.
How the three layers stack
The layers are strictly nested. The harness contains the context layer, which contains the prompt.
- Harness engineering wraps everything: orchestration, guardrails and approvals, tracing and evals, deployment and policy boundaries.
- Context engineering sits inside it: retrieval, memory and compaction, tool visibility, session state and history.
- Prompt engineering sits at the core: instructions, examples, output schema, formatting and roles.
- Context engineering sits inside it: retrieval, memory and compaction, tool visibility, session state and history.
The production architecture that current OpenAI, Anthropic, MCP, and LangGraph guidance all point at looks like this in sequence: a user or upstream system hits an orchestrator; the orchestrator assembles context from a prompt builder, a retrieval layer, a memory layer, and a tool registry; a context assembler feeds the model; the model makes tool calls; validators and policy checks gate the output; a human approves anything destructive or high-cost; the action fires; and every step (the model call, the tool calls, the validation, the approval, the final action) streams into tracing, which feeds offline and online evals. Retrieval, memory, tools, validation, and logging belong around the model, not inside a single prompt.
The three layers side by side
| Discipline | Scope | Typical artifacts | Common failure modes |
|---|---|---|---|
| Prompt engineering | Instruction design for one model call or bounded exchange. Wording, role, examples, output format. | Prompt templates, example libraries, schema definitions, prompt caches. | Ambiguity, brittleness across models, overfitting, injection through mixed trusted and untrusted text. |
| Context engineering | Selection and maintenance of the right information and tools per turn. Token-budget quality. | Indexes, retrievers, compaction rules, memory stores, subagent configs, context policies. | Context rot, stale or irrelevant retrieval, memory poisoning, long-context dilution, tool confusion. |
| Harness engineering | Full production runtime around the model. Orchestration, validation, approvals, tracing, governance. | Orchestrators, runbooks, handoff maps, guardrails, approval flows, traces, regression suites. | Tool misuse, approval deadlock, hidden state bugs, poor observability, insecure integrations, regression after updates. |
The same three layers, three products
For invoice processing, prompt engineering defines the extraction instruction, context engineering supplies the supplier record, PO, tax policy, and prior decisions, and harness engineering adds duplicate detection, exception routing, approval pauses, ERP posting, and audit logs. A “great prompt” still fails if the supplier history is absent or the posting action is unguarded.
For a coding agent, prompts set style, testing expectations, and output conventions; context exposes the relevant repo docs, files, tests, and recent changes; the harness governs branch workflows, test execution, CI, traces, approvals, and benchmarked regression checks such as SWE-bench-style validation.
For a customer-support chatbot, prompts control brand voice, refusal behaviour, and answer format; context handles document retrieval, session memory, and tool selection; the harness manages escalation, human review for sensitive outputs, online evals, and monitoring.
Adjacent terms worth knowing
The vocabulary is still stabilising, so treat these as branches of the three core layers rather than separate disciplines:
- Instruction engineering. Structuring and prioritising instructions across roles (developer vs user vs third-party text). A safety-adjacent subfield of prompting.
- Retrieval engineering. Indexing, chunking, ranking, reranking, relevance evaluation. A major branch of context engineering.
- Memory engineering. Short-term and long-term memory, note-taking, compaction, recall policies. Context engineering with some harness overlap.
- Tool-use engineering. Tool schemas, descriptions, permissions, and response shapes models can reliably choose. Between context and harness.
- Agent orchestration. Which agents run, in what order, via LLM choice or code. Core harness.
- Guardrail and safety engineering. Input, output, and tool checks; approval flows; injection defence. Primarily harness.
- Observability engineering. Traces, logs, costs, latencies, run-level events. A harness capability.
- Evaluation engineering. Datasets, offline tests, online monitors, judges, regression suites. Cross-cutting, decisive in the harness.
A build order for a mid-size enterprise
The highest-return sequence is consistent across current vendor guidance, and it is the one I use with clients.
First, standardise bounded use cases. Start with narrow tasks where correctness can be judged deterministically or with targeted human review: extraction, classification, summarisation with references, guided support responses. Build prompt baselines and task-specific evals immediately. Do not jump to high-autonomy agents before you have reference cases and instrumentation.
Next, build the context layer before broad autonomy. Add retrieval, document indexing, and memory only where they measurably improve outcomes. Curate minimal toolsets and define compaction policies up front. For most enterprises the first real value jump comes not from smarter prompting but from connecting models to accurate internal data through disciplined retrieval and memory, which is exactly where MCP earns its keep.
Then, implement the harness. Introduce orchestration, tracing, guardrails, approval boundaries, and online monitoring. Make non-destructive tasks autonomous first, and keep destructive, external, or high-cost actions behind explicit approval until traces and evals give you the evidence to loosen them.
Finally, optimise and specialise. Once the system is instrumented, use prompt optimisation, specialist subagents, routing logic, and cheaper model tiers where the measurements support it. DSPy-style optimisers and subagent isolation get far more valuable once you already know what “good” looks like.
In short: make the model understandable, then informed, then dependable.
The bottom line
Prompt engineering remains necessary. Context engineering is increasingly decisive. Harness engineering is what separates a demo from a production system. The centre of gravity in AI engineering has moved from finding the right words to building the right system around the model, and every serious vendor’s documentation, every mature agent runtime, and the recent research literature all point the same way.
If you want help mapping your own use cases onto these three layers, or building the harness that makes agents safe to ship, the way I work with organisations is built around exactly this. For the platform-level view of what to build on, see The Best AI Agent Platforms and Frameworks of 2026.