Software Engineering at Google
Lessons Learned from Programming Over Time
by Titus Winters, Tom Manshreck & Hyrum Wright
Buy on Amazon"Software engineering can be thought of as 'programming integrated over time'" — this book covers how Google manages a codebase of more than two billion lines of code with 50,000 engineers, organized around three principles: time and change, scale and growth, and trade-offs and costs.
What This Book Is About
The preface asks what distinguishes "software engineering" from "programming." The book's answer: time, scale, and the trade-offs at play. Programming is about producing code. Software engineering adds development, modification, and maintenance over the life of that code. The expected life span of code varies by a factor of roughly 100,000 — from minutes to decades.
The book defines sustainability: "Your project is sustainable if, for the expected life span of your software, you are capable of reacting to whatever valuable change comes along, for either technical or business reasons." It covers Google's culture (Part II), processes (Part III), and tools (Part IV). The preface is clear this is not a sermon: "The lessons that we have learned, we learned through our failures."
The Laws and Rules
The book introduces several named principles that Google uses internally:
Hyrum's Law — "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody." Named after co-author Hyrum Wright, who "tried really hard to humbly call this 'The Law of Implicit Dependencies,'" but "Hyrum's Law" is the shorthand most people at Google settled on.
The Beyonce Rule — "If you liked it, you should have put a CI test on it." If infrastructure changes pass all your tests but break your product, you are on the hook for fixing it.
The Churn Rule — Introduced in 2012. Infrastructure teams must do the work to migrate their internal users to new versions, or do the update in place in a backward-compatible way. This scales better than pushing migration work to every consumer.
Shifting Left — Finding problems earlier in the developer workflow is cheaper. "Bugs that are caught by static analysis and code review before they are committed are much cheaper than bugs that make it to production."
The One-Version Rule — "Developers must never have a choice of 'What version of this component should I depend upon?'" Only one version of each external dependency is allowed in the repository.
Testing (Chapters 11-14)
The testing chapters are the most detailed in the book. The pivotal story: in 2005, Google Web Server (GWS) had more than 80% of production pushes containing user-affecting bugs that had to be rolled back. After instituting automated testing for all new code, emergency pushes dropped by half within a year.
1 Test Sizes
Google classifies tests by size (resource constraints), not just scope:
Small
Single process, often single thread. No I/O, no network, no disk, no sleep. Google runs Java small tests with a security manager that fails the test on prohibited actions.
Medium
Can span multiple processes. Can use threads, blocking calls, and network calls to localhost only. Good for running a local database or browser via WebDriver.
Large
No restrictions. Can span multiple machines. For full end-to-end and system tests. Default timeouts of 15 minutes to 1 hour.
2 The Test Pyramid
The recommended ratio: ~80% unit tests, ~15% integration tests, ~5% end-to-end tests. Two antipatterns the book names: the "ice cream cone" (too many end-to-end, too few unit) and the "hourglass" (many unit and end-to-end but few integration tests).
3 Unit Testing Rules
Test via public APIs, not internal details. The book says a test that calls private methods
like isValid() or saveToDatabase() is brittle because renaming or refactoring breaks it,
even though users wouldn't notice.
Test state, not interactions. State testing checks what happened after calling a method. Interaction testing checks how the method was called. The book says over-reliance on mocking frameworks is "the most common cause of problematic interaction tests."
Prefer real objects over test doubles. "At Google, we have found that [mockist testing] style of testing is difficult to scale." Google's codebase "has suffered so badly from an abuse of mocking frameworks that it has led some engineers to declare 'no more mocks!'"
DAMP, not DRY. Test code should be "Descriptive And Meaningful Phrases." Some duplication is OK if it makes tests simpler and clearer. DRY is for production code; tests should prioritize readability.
4 Larger Testing and Configuration
The book states directly: "At Google, configuration changes are the number one reason for our major outages." A 2013 global Google outage was caused by a bad network configuration push that was never tested. Unit tests are described as "like a problem in theoretical physics: ensconced in a vacuum, neatly hidden from the mess of the real world."
Code Review (Chapter 9)
At Google, every change is reviewed before commit. Every change needs three approval "bits": correctness (LGTM from a peer), code ownership (from a directory owner via OWNERS files), and readability (from someone certified in that language's style). In practice, one person often fills all three roles.
Keep changes small. About 35% of changes at Google are to a single file. Most are about 200 lines. Most reviews are done by one reviewer. Initial feedback is expected within about a day.
"The customer is always right." If a reviewer asks a comprehension question, that question will be multiplied many-fold over time. Any confusion means the code isn't clear enough.
"Code is a liability." The book says: "If you're writing it from scratch, you're doing it wrong!" Research should be done before writing new code.
Culture (Chapters 2-7)
Three Pillars of Social Interaction
- Humility — You are not the center of the universe
- Respect — Genuinely care about others
- Trust — Believe others are competent
The book claims: "If you perform a root-cause analysis on almost any social conflict, you can ultimately trace it back to a lack of humility, respect, and/or trust."
Three "Always" of Leadership
- Always Be Deciding — Identify blinders, identify trade-offs, decide and iterate
- Always Be Leaving — Get your organization to solve problems by itself, without you present
- Always Be Scaling — Protect your time, attention, and energy
The book's litmus test: "Think about the last vacation you took that was at least a week long. Did you keep checking your work email? (Most leaders do.) Ask yourself why."
Google's own research found that psychological safety is the most important part of an effective team. Google X (moonshot division) rewards people for how many ideas they can disprove or invalidate. The book says: "Failure is an option" is a favorite Google motto. Blameless postmortems must contain: summary, timeline, primary cause, impact and damage assessment, action items (with owners) to fix immediately, action items to prevent recurrence, and lessons learned.
Tools and Scale
Trunk-based development. The book argues dev branches are "inherently misguided." Google's approach: commit to trunk, rely on testing and CI, disable incomplete features at runtime. "There is a predictive relationship between trunk-based development and high-performing software organizations."
Large-Scale Changes (LSCs). Google has infrastructure for automated codebase-wide changes. The largest series "removed more than one billion lines of code over the course of three days." At peak throughput during one migration: 700+ independent changes, 15,000+ files per day.
Static analysis via Tricorder. Analyzes more than 50,000 code review changes per day. Includes more than 100 analyzers across 30+ languages. Overall effective false-positive rate: just below 5%. Authors apply automated fixes about 3,000 times per day.
CI at scale (TAP). Google's Test Automation Platform handles more than 50,000 unique changes and runs more than four billion individual test cases every day. Average wait to submit: about 11 minutes. A change that passes presubmit has a "very high likelihood (95%+)" of passing all remaining tests. Any change can be rolled back with two clicks.
Deprecation. "Code is a liability, not an asset." The book says: "Code itself doesn't bring value: it is the functionality that it provides that brings value." Compulsory deprecations should be staffed by a specialized team, not pushed to consumers. Google uses planned outages of increasing duration before final turndown to discover unknown dependencies.
10 Things You Can Apply at Work
Test via public APIs, not implementation details. The book says after writing a test, "you should never need to touch it again" unless requirements change. Tests that call private methods break on refactoring.
Prefer real objects over mocks. Use a real implementation when it's fast, deterministic, and has simple dependencies. The book says fakes (lightweight implementations) are preferred over stubs and mocks when real objects won't work.
Classify tests by size (small/medium/large), not just scope. Small tests run in one process with no I/O. Medium tests can use localhost. Large tests have no restrictions. This prevents flaky tests from slow or nondeterministic resources.
Keep code reviews small (~200 lines) and fast (~24 hours). Default to one reviewer. Require three types of approval: correctness, ownership, and readability. Automate formatting and linting via presubmit.
Treat documentation like code. The book says to put docs under source control, assign ownership, require reviews, track issues as bugs, and attach freshness dates with review reminders.
Use the Beyonce Rule for your CI. If infrastructure changes break your product but not your tests, that's your problem. Test performance, correctness, accessibility, and failure handling.
Enforce style guides with automated tools, not humans. The book says roughly 90% of Google's C++ style guide rules could be automatically verified. Presubmit checks reject non-compliant code. "The robots are better on average than the humans by a significant amount."
Write blameless postmortems. The book requires seven elements: summary, timeline, primary cause, impact and damage assessment, action items (with owners) to fix immediately, action items to prevent recurrence, and lessons learned. "Don't erase your tracks—light them up like a runway for those who follow you!"
Version-control your configuration alongside code. The book says "a large percentage of production bugs are caused by 'silly' configuration problems" and that "configuration changes are the number one reason for our major outages."
Before measuring anything, ask: "If we get a negative result, will appropriate action be taken?" The book says this question "stops most of the projects that our research team takes on." The book also warns that using productivity metrics for individual performance reviews is counterproductive — "engineers will be quick to game the metrics."
Quotes from the Book
"Software engineering is programming integrated over time."
"It's programming if 'clever' is a compliment, but it's software engineering if 'clever' is an accusation."
"With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody."
"'Because I said so' is a terrible reason to do things."
"Code itself doesn't bring value: it is the functionality that it provides that brings value."
Who Should Read This
- SDETs and test engineers — Chapters 11-14 on testing are the most detailed treatment of testing at scale in any book. The test size definitions, test doubles chapter, and larger testing strategies are directly usable.
- Engineering leads and managers — The culture chapters (leadership, knowledge sharing, measuring productivity) and process chapters (code review, style guides, documentation) provide concrete practices backed by Google-scale evidence.
- DevOps and infrastructure engineers — The tools chapters cover version control, build systems, CI/CD, static analysis, and large-scale changes with specific numbers and architecture details from Google's internal systems.
- Anyone maintaining a long-lived codebase — The book's core thesis — that sustainability over time is the defining challenge of software engineering — applies to any project expected to last more than a few years.
§ Verdict
8 / 10
The testing chapters (11-14) and the chapter on deprecation (15) are the strongest parts of the book. The culture chapters (2-7) are useful but repeat some ideas across chapters. The tools chapters (16-25) are fascinating as a window into Google's infrastructure but some are very Google-specific (Code Search, Critique, Borg). The book is honest about Google's own failures, which makes the advice more credible. At 25 chapters it's long — if you only have time for a few, read chapters 1 (the thesis), 11-13 (testing), and 15 (deprecation).