Wally Stegner: Systems Architect
Wallace Stegner's Angle of Repose opens with a historian named Lyman Ward, wheelchair-bound and estranged from his present, attempting to reconstruct the life of his pioneer grandmother, Susan Burling Ward, from her letters and journals. He has the artifacts — the primary sources — but not the context. He knows what she wrote, but not always what she meant. He can see the shape of events but cannot always explain why decisions were made, or what forces were quietly at work beneath the surface. If you've spent any real time in data engineering, this probably sounds familiar.
The title of Stegner's novel comes from geology. The angle of repose is the steepest angle at which a sloping surface of loose material — gravel, sand, talus — remains stable. Exceed it, and things slide. The material doesn't fail catastrophically all at once; it settles, shifts, finds a new equilibrium. Or it doesn't.
Data systems have an angle of repose too. Every pipeline, every warehouse, every reporting layer exists in a kind of provisional stability. It works — until the business changes faster than the data model can accommodate. Until the source system gets upgraded and the field names shift. Until someone adds a new product line that doesn't fit the existing dimensional schema. The system doesn't announce its instability. It just starts to slide: reports go slightly wrong, reconciliations drift, someone in Finance notices a number that doesn't match and sends an email with three question marks in the subject line.
The question data engineers live with — the one that rarely makes it into job descriptions — is not can we build this, but at what angle is this stable? How much change can this model absorb before it needs to be rethought? How much technical debt is load-bearing, and how much is just sediment?
Lyman's reconstruction project is an act of institutional archaeology. Susan left behind thousands of letters, but letters are not documentation. They are artifacts of a moment, written for an audience of one, shaped by what the writer chose to disclose. Lyman has to infer the rest — the business logic, you might say, behind the recorded transactions.
This is precisely the condition of inheriting a legacy data environment.
Every organization has a Susan Ward: some original architect, or more likely a succession of them, who built something that worked in the context of their moment. The decisions they made were rational given what they knew. The exceptions they hardcoded, the surrogate keys they chose, the ETL jobs that run at 2 a.m. for reasons no one alive can fully explain — all of it made sense once. The letters exist. The meaning is harder.
What gets lost is not the data. It's the why. The institutional knowledge that explains why the fiscal calendar starts in October, why certain cost centers are excluded from the departmental rollup, why that one field has three different null conventions depending on the source system. This is what I've come to think of as the real informational moat in data work — not SQL fluency, not pipeline tooling, but the accumulated understanding of why the data is the way it is. Lyman can read Susan's letters. Understanding them is a different project entirely.
Susan Burling Ward was a pioneer in the literal sense — she followed her engineer husband into the frontier West, to mining camps in California and irrigation projects in Idaho, places where infrastructure was being invented in real time and failure was a condition of the work, not an exception to it. She built a life in unsettled terrain and she was good at it, even when it cost her.
Data engineering is still frontier work, even if the frontier now lives in cloud console dashboards and YAML configuration files. The tooling changes faster than the practices. Standards emerge, get adopted, get deprecated. What was best practice two years ago is a cautionary tale today. Every team building a modern data stack is, to some degree, making it up as they go — reading what others have done, adapting it to terrain that doesn't quite match, hoping the angle holds. This is not a complaint. The frontier quality of the work is part of what makes it interesting. But it does mean that the pioneering metaphor comes with its costs. Susan sacrificed stability for the sake of her husband's ambitions; data teams sacrifice sustainability for the sake of delivery timelines. The pipeline ships, the dashboard goes live, the backlog moves on — and the decisions that were made under pressure become the sediment that the next team inherits. Stegner is clear-eyed about this. He doesn't romanticize Susan's pioneer life. He sees both what it built and what it cost. Good data engineering requires the same dual vision: an honest accounting of what a system does well and what it's quietly mortgaging against the future.
Lyman ends his reconstruction with something less than certainty. He understands Susan better than when he started, but the gaps remain. Some letters were lost. Some decisions are simply unrecoverable. He makes his peace with that, and keeps writing.
This too is the condition of the data engineer. You will not always find the original logic. Some of the source system documentation was never written. Some of the people who knew are gone. You do the archaeology you can, you document what you find, you leave the next person better positioned than you were — and you accept that the map will always be incomplete.
The angle of repose is not a permanent state. It's a negotiation between the material and the slope, repeated continuously, adjusted as conditions change. The goal isn't to find a system that never slides. It's to build something stable enough to do its work, with enough structural awareness that when it does shift, you understand why — and you can begin, again, to reconstruct.
Coauthored with Claude