Context Engineering vs Prompt Engineering: Karpathy's Method

Somewhere right now an SDET is rewriting the same prompt for the fifth time, convinced the right wording will finally make Copilot stop inlining CSS selectors. Andrej Karpathy settled the context engineering vs prompt engineering debate back in June 2025, and his verdict says that engineer is tuning the wrong layer entirely: the prompt was never the problem. What the model knows when it answers is.

Karpathy has better credentials for this call than almost anyone — OpenAI founding member, former Director of AI at Tesla, and the educator behind more than six hours of YouTube instruction on how LLMs actually work. He’s also the person who coined “vibe coding,” then spent 2025 publicly walking back its scope. By February 2026 he was calling agent-driven development “agentic engineering” — “an art & science and expertise… something you can learn and become better at.” If you’re an SDET reviewing AI-generated tests every day, that expertise is now part of your job description. This post distills his method from the primary sources — the tweets, the talks, the 3.5-hour deep dives — and maps it onto test automation, where I think it lands harder than anywhere else.

Context engineering vs prompt engineering: what did Karpathy say?

On June 25, 2025, Karpathy posted “+1 for ‘context engineering’ over ‘prompt engineering’” on X, defining it as “the delicate art and science of filling the context window with just the right information for the next step.” The post drew over 14,000 likes and effectively renamed the discipline — a year on, “context engineering” is the term you’ll meet in framework docs and job postings.

His reasoning: people associate prompts with the short task descriptions you type into a chat box. But in any industrial-strength LLM application, the prompt is a small slice of what actually enters the model. The rest is engineered context — task descriptions and explanations, few-shot examples, retrieved documents, multimodal data, tool definitions, state and history, compaction of all of the above.

That second failure mode is the one teams miss. Researchers now call it context rot — model recall degrades as the window fills with irrelevant tokens. A bigger context window is not a license to paste everything; it’s a bigger surface to engineer.

Why do LLMs need context engineering?

Because an LLM’s weights are, in Karpathy’s words, “a zip file of the internet” — a lossy, roughly 100x compression of its training data. Parameters give the model a vague recollection of everything and a reliable memory of almost nothing. The context window is the only memory you control, and his entire method follows from treating it as the scarce resource it is.

Across his Software 3.0 keynote and the Deep Dive into LLMs like ChatGPT video, he cataloged the “psychology” of these models — and every trait maps directly to a working rule:

The amnesia point deserves emphasis because it’s the one with the biggest workflow implication. A human colleague consolidates knowledge — they learn your framework once. An LLM’s working memory, as Karpathy puts it, “gets wiped every single morning,” like the protagonist of Memento. You don’t fix that by explaining your conventions again in chat number forty-seven. You fix it by writing the explanation down once, in a file the tool loads on every invocation. (The gullibility row is its own topic — I covered the CI implications in prompt injection through Playwright MCP.)

Karpathy’s method, distilled

Across “How I use LLMs” (2h11m, February 2025), “Deep Dive into LLMs like ChatGPT” (3h31m), and the Software 3.0 keynote, the same handful of rules keep resurfacing.

Own the window

Karpathy starts a new chat for every topic switch. Stale tokens from the previous task distract the model, degrade accuracy, and make every subsequent token slower and more expensive to process. He pastes source material directly into the window instead of trusting recall — a book chapter, an API doc, a stack trace — and uploads documents rather than summarizing them. For images with data that matters (his example: a blood test), he runs a two-step flow: transcribe to text first, verify the transcription, then analyze. Generation you can check beats analysis you can’t.

Show, don’t describe

For any repeatable task, Karpathy gives the model concrete input/output examples rather than describing the task — few-shot prompting, delimited with XML-style tags so the model knows exactly where the example ends and the work begins. For a test-generation agent, that looks like this:

# Test conventions
- Locators: getByTestId or role-based only — never CSS selectors
- Every test imports fixtures from `src/fixtures`; never inline auth
- Assert on business values (totals, names), not just visibility

<example>
test('checkout shows the discounted total', async ({ checkoutPage }) => {
  await checkoutPage.applyCoupon('SAVE20');

  await expect(checkoutPage.orderTotal).toHaveText('$103.20');
  await expect(checkoutPage.discountBadge).toHaveText('-20%');
});
</example>

One committed example does more work than a paragraph of adjectives, because the model is a pattern-completion machine: it will clone the shape of what it sees. That’s also why a vague prompt fails in a specific way — Karpathy’s keynote framing is that vague prompts fail verification, which sends you back to re-prompt, and “you’re going to start spinning.” Spending an extra minute making the request concrete is cheaper than three round trips.

Buy computation, don’t beg for it

“Models need tokens to think” is the most mechanically grounded rule in the set: a transformer performs a finite amount of computation per token, so an answer demanded in one token gets one token’s worth of thought. Ask for the setup, the intermediate steps, the calculation — you’re literally buying compute with output tokens. For anything involving arithmetic, counting, or exact string manipulation, Karpathy adds “use code” to the prompt, forcing a Python interpreter call instead of trusting what he calls the model’s mental arithmetic. And he reserves thinking models for genuinely hard problems — difficult bugs, math, deep logic — while defaulting to fast models for 80–90% of his queries. For anything high-stakes, he runs an “LLM council”: the same prompt across several providers, answers compared. That treat-tokens-as-money instinct scales straight to CI, which is why I put a token budget on Playwright AI agents before letting them loop.

What does context engineering mean for test automation?

Every AI testing tool you touch is a context pipeline, and almost every “the AI wrote a bad test” complaint is a context failure upstream of the prompt. That’s not a metaphor — it’s literally how the tools are built.

GitHub Copilot reads committed instruction files on every interaction; I walked through the three files that fixed our enterprise setup in Your Copilot Isn’t Dumb — You’re Starving It of Context. Playwright’s generator agent assembles its context from an agent definition, a seed spec, and the plan — engineer those three files and quality stops being a coin flip, as I showed in Your Playwright Generator Is Only as Good as Its Seed. I learned the cost of skipping this the slow way: on one enterprise suite, a single convention the generator never saw multiplied across 300 generated specs into a quarter of cleanup. The fix wasn’t a better prompt — it was a seed file. Karpathy’s method is the theory those posts are the practice of.

There’s a sharper enterprise point hiding in his October 2025 interview with Dwarkesh Patel. Karpathy hand-wrote his ~8,000-line nanochat project and found coding agents “net unhelpful” on it — his diagnosis: “possibly the repo is too far off the data distribution.” He also complained that models kept pushing him back toward standard PyTorch patterns he’d deliberately avoided. Sound familiar? Your in-house test framework — the custom fixtures, the wrapper around Playwright, the naming conventions a decade of enterprise scar tissue produced — is off-distribution in exactly the same way.

And the verification half of his method should feel like home. Karpathy’s core loop is “AI does the generation, the human does the verification” — and verification is the skill SDETs have been sharpening for decades. A recent Ask HN thread on agentic coding is full of engineers discovering what we already knew: agents happily produce tests that pass while asserting nothing — one commenter’s example was the expect(true).to.be(true) genre of test. Catching that is a solved problem in QA — mutation testing exposes fake AI tests mechanically, and keeping the model in a triage role rather than a deciding role — the autonomy-slider position I argued for in Let AI Triage Your Playwright Failures, Not Decide Them — is exactly Karpathy’s “partial autonomy” applied to CI.

Keep the AI on a leash — where Karpathy landed in 2026

Karpathy’s 2025 keynote advice was to keep the AI “on the leash”: small incremental chunks, single concrete tasks, a generation-verification loop spinning as fast as possible, and no 10,000-line diffs — because the human reviewer is the bottleneck, and an unreviewable diff is a liability with a green checkmark. Watch his trajectory since, though, because the leash length changed while the loop didn’t. February 2025: vibe coding, scoped in his own retrospective to “fun throwaway projects.” October 2025: calling industry agent hype “slop” and preferring autocomplete for expert work — while still copy-pasting his entire repo into GPT-5 Pro as an oracle for hard questions, which is context maximalism in its purest form. February 2026: agentic engineering as the professional default, orchestrating agents “99% of the time” with oversight.

That’s why I’d argue his method matters more for SDETs than for anyone else: we’re the discipline that industrialized verification, and that’s exactly where the SDET role is heading in the age of AI. The teams winning with AI test generation aren’t the ones with the cleverest prompts — they’re the ones with committed context files and merciless review gates.

Your next step: write the context file

Pick the repo where AI writes tests for you and give it a memory this week. One file is enough to start: CLAUDE.md, .github/copilot-instructions.md, or Playwright’s agent definition plus seed spec — whatever your tool reads on every run. Put three things in it: your locator standard, your fixture imports, and one complete example test inside <example> tags. Then delete the paragraph of instructions you’ve been re-pasting into chat, and let the file do the remembering — the model can’t consolidate knowledge, but your repo can.

If you want the full implementation walkthroughs, start with the Copilot instruction files and then the Playwright generator seed.